Model routing cuts agent cost by letting a small or on-device model handle routine steps and handing only uncertain steps to a frontier model. Published work shows this can hold quality while spending less. One weak-to-strong framework matched an always-on frontier agent's task success at 19.2% lower inference cost. The saving is not automatic. It depends on a trustworthy signal for when to escalate, and on an eval gate that blocks any routing change that lets quality drop.
Why route at all
No single model is best at every task once cost is part of the question. Routing systems exist to combine the strengths of several models. The price gap is large: FrugalGPT reported that LLM API fees can differ by two orders of magnitude. Agents make this worse. Anthropic notes that agentic systems often trade latency and cost for better task performance, and that autonomy means higher costs and potentially compounding errors. Every step an agent takes is a model call, so sending all of them to the most expensive model is the costliest default.
Three routing patterns
- Up-front routing: classify the input, then direct it to a specialized path. Anthropic describes this as working when categories are distinct and a classifier (an LLM or a traditional one) can sort them accurately. Its cost example sends easy or common questions to smaller models and hard or unusual ones to more capable models.
- Cascades with a deferral rule: a smaller model decides whether it can handle the query or must pass it to a larger, costlier model, so the large model is used only for complex work. FrugalGPT is a cascade that learns which model combinations to use per query.
- Weak-to-strong escalation inside an agent: a weak agent acts by default and hands control to a strong one only when needed. The hard part is deciding when, which needs a cheap per-step uncertainty signal.
What the published results show
- LEASH on OSWorld (computer-use agents)
- 19.2% lower cost
- 73.66% task success vs 72.90% for always-on Claude Opus 5
- FrugalGPT cascade
- up to 98% cost reduction
- while matching the best single LLM, e.g. GPT-4
- Hybrid LLM router
- up to 40% fewer large-model calls
- with no drop in response quality
- RouteLLM
- over 2x cost reduction
- in certain cases, without hurting response quality
These are different benchmarks, model pairs and task types, so the figures are not comparable with each other. They show that routing can work, not what it will save on your workload. The word 'up to' in several of them matters.
Deciding when to escalate
The escalation signal decides whether routing pays off. LEASH compares the weak model's hidden states with those from past successful and failed trajectories. It reuses activations already computed during action generation, so it needs no extra forward pass and no dedicated monitor training. At matched escalation rates it beat random escalation by 6.55 to 11.94 points on OSWorld across four model backbones. That margin is the value of a good signal.
Signals based on correctness can need costly ground-truth supervision in open-ended and agentic tasks. Disagreement-guided routing avoids this by using model-to-model disagreement, measured directly from generated responses with no ground-truth labels, and it can even beat methods trained to predict correctness. Its approximation improves as the strong model becomes a better proxy for the ground truth. Other options are a confidence check, a comparative check, a cost-benefit analysis, or a token-specific check against the large model's top-ranked tokens. Hybrid LLM adds a tunable quality level that can be changed at test time to trade quality for cost.
Where the small or on-device model fits
Small models are now cheap and fast enough to be the default tier. Liquid AI's LFM2.5-24B-A2B is a 24B-parameter mixture-of-experts model with 2B active parameters. It decodes at 184 tokens/s on an Apple M5 Max and is priced at $0.10 per 1M input tokens and $1.00 per 1M output tokens. Liquid AI also reports that larger models keep an edge on knowledge-heavy and coding benchmarks. That gap is the reason to keep an escalation path rather than run the small model alone.
The eval gate that stops quality dropping
A router is a model-quality decision made automatically, so it needs a gate like any other change. A self-hosted open-source agent platform shows the shape: a small golden set scores every model change by rule, which is repeatable and uses no LLM judge, and it has a quality gate, an audit trail and one-click rollback. In our builds we apply the same idea to routing thresholds, not only to model swaps.
Build a golden set per step type
Collect real examples of each kind of agent step, with the expected outcome, so routine and hard steps are both represented.
Score by rule where you can
Prefer checks that are repeatable and need no LLM judge. Reserve judged scoring for outputs that cannot be checked by rule, and state that limit.
Set the escalation threshold against the gate
Because the quality level can be tuned at test time, treat the threshold as a setting. Pick the cheapest value that still passes the gate.
Gate every change
A new small model, a new frontier model, a prompt edit or a threshold change all rerun the golden set before release.
Keep an audit trail and rollback
Log which tier handled each step and keep a one-step path back to the previous configuration.
For comparing routing strategies themselves, RouterBench provides an evaluation framework with over 405k inference outcomes from representative LLMs. It is useful for choosing a strategy; it does not replace a gate built on your own tasks.
Putting a gateway in front
A gateway keeps routing out of application code. The open-source platform above uses LiteLLM in front of Ollama, offering one OpenAI-compatible API and model aliases for every backend, with a model chosen per use case such as chat, agent or e-mail. Swapping the model behind an alias then becomes a gated configuration change instead of a code release.
Failure modes of routing
- Weak escalation signal: random escalation is the baseline, and a good signal beat it by 6.55 to 11.94 points at matched rates. A poor signal gives up most of that gain.
- Confidently wrong small model: a deferral rule based on the small model's own confidence only works if that confidence is trustworthy. The gate should test for this directly.
- Weak proxy for truth: disagreement-based routing gets more accurate as the strong model becomes a better proxy for the ground truth, so a strong model that is itself wrong undermines it.
- Wasted time in sequential cascades: if the small model is not confident, the wait is lost and the large model starts from scratch. Speculative cascades address this with a token-by-token deferral rule.
- Knowledge-heavy and coding work: larger models keep an edge here, so these steps may need to default to the strong tier.
- Compounding errors: in an agent, a wrong early step carries into later ones, so a cheap mistake can become an expensive one.
- Model-pair drift: RouteLLM's routers kept their performance when the model pair changed, but that is a benchmark result. Rerun the gate after any change.
What stays with a person
Routing decides which model acts; it does not decide which actions are safe to take unattended. Steps that move money, change records or delete data should keep human approval or hard guardrails whichever tier handled them. Cost should not be the reason to loosen those controls.
- production-ai-agent-development
- what-is-a-production-grade-ai-agent
- what-must-stay-human-approved-in-ai-bookkeeping
- how-to-stop-ai-agent-deleting-data
Sources
- [2610.22280v1] LEASH: Uncertainty-Aware Weak-to-Strong Escalation and Oversight for Computer-Use Agents: http://arxiv.org/abs/2610.22280v1
- [2610.22245v1] Disagreement-Guided Weak-to-Strong Routing: http://arxiv.org/abs/2610.22245v1
- [2406.18665] RouteLLM: Learning to Route LLMs with Preference Data: https://arxiv.org/abs/2406.18665
- [2305.05176] FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance: https://arxiv.org/abs/2305.05176
- [2404.14618] Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing: https://arxiv.org/abs/2404.14618
- [2403.12031] RouterBench: A Benchmark for Multi-LLM Routing System: https://arxiv.org/abs/2403.12031
- Building Effective AI Agents \ Anthropic: https://www.anthropic.com/engineering/building-effective-agents
- Speculative cascades — A hybrid approach for smarter, faster LLM inference: https://research.google/blog/speculative-cascades-a-hybrid-approach-for-smarter-faster-llm-inference