)
What is LLM model routing?
LLM model routing is the practice of automatically directing each query to the most cost-effective model capable of answering it well. Simple tasks go to cheap, fast models. Complex ones go to frontier models. Done right, routing cuts inference costs by 40-85% while keeping output quality near-identical to always using the most expensive model.
TL;DR - Key Takeaways
- GPT-4o costs 16x more per token than GPT-4o mini. For most queries in production, the cheaper model performs identically
- RouteLLM (UC Berkeley, ICLR 2025) achieved over 85% cost reduction on MT Bench while maintaining 95% of GPT-4 performance, sending only 14% of queries to the strong model
- FrugalGPT (Stanford, 2023) demonstrated up to 98% cost reduction through cascade routing, where cheap models handle most queries and expensive ones only see the hard ones
- The three practical routing strategies are: classifier-based routing, cascade routing, and semantic routing, each suited to different workloads
- Routing logic is most durable when enforced at the gateway layer, not hardcoded into individual applications
Most teams send everything to GPT-4o because it is the safe default. That is like hiring a neurosurgeon to take your temperature.
The neurosurgeon is excellent. You just paid $400 for a 10-second job. LLM model routing fixes this by matching query complexity to model capability automatically.
This guide covers the three strategies that actually work in production and the tools you can use to implement them today. For the full cost optimization framework, read the AI Token Optimization Complete Guide.
)
The Problem With Defaulting to the Best
Here is something I see constantly. A team picks GPT-4o for their first AI feature because it is the best. It works. They ship more features using the same model. Nobody stops to ask whether every query actually needs it.
Six months later, 80% of their inference budget is going to a frontier model that is answering things like "summarise this in two sentences" and "is this email urgent, yes or no?"
GPT-4o costs $2.50 per million input tokens. GPT-4o mini costs $0.15. That is a 16x difference. Claude 3 Haiku sits around $0.80. Gemini 1.5 Flash at $0.075 is 33x cheaper than GPT-4o.
The output quality gap on simple, structured tasks? Near zero.
You are not paying for better answers. You are paying for capability you are not using.
LLM model routing is the fix. Instead of a single model for everything, you build a routing layer that looks at each incoming query and sends it to the right model / cheapest model capable of doing the job well.
)
The Three Routing Strategies
Strategy 1: Classifier-Based Routing
You train a lightweight classifier (a small BERT model or a fast LLM prompt) to predict query complexity before the query reaches your main model. The classifier outputs a score or category. Queries below a threshold go to the small model. Queries above go to the frontier model.
This is the approach RouteLLM uses. The Berkeley and Anyscale research team trained routers on preference data from the LMSYS Chatbot Arena, pairs of responses where humans judged which model did better. The routers learned, from that signal, which queries actually need the strong model.
Their results, published at ICLR 2025:
- Matrix factorization router: 85% cost reduction on MT Bench, 95% of GPT-4 performance, only 14% of queries sent to the strong model
- BERT-based classifier: 45% cost savings on MMLU at comparable quality
- With LLM-judge data augmentation: same 95% quality with only 14% strong-model calls, 75% overall cost reduction Classifier-based routing works best when your query distribution is relatively predictable. Classification tasks, yes/no questions, structured extraction, and summarisation are almost always fine on small models. Deep reasoning, open-ended generation, and multi-step code are not.
Best for: Teams with high query volume and identifiable simple/complex split. Low latency overhead once the classifier is trained.
Strategy 2: Cascade Routing
Cascade routing does not pre-classify queries. It sends every query to the cheap model first, checks whether the response meets a confidence threshold, and only escalates to the expensive model when it does not.
This is the core idea behind FrugalGPT, the Stanford paper from Lingjiao Chen, Matei Zaharia, and James Zou. The cascade tries cheaper models in sequence, stops when one produces a confident-enough answer, and only calls the expensive model if all others fail the threshold.
Stanford's results: up to 98% cost reduction compared to always using the best LLM API, with the same output quality. On average across benchmarks, FrugalGPT-style cascades achieved 50-98% cost savings.
The mechanism that makes cascades work is the confidence check. For structured tasks where the model knows whether it knows the answer this is reliable. For open-ended generation, where models are confidently wrong regularly, cascades are trickier. You need a quality judge (another model or a scoring function) rather than a raw confidence score.
Best for: Workloads where the cheap model handles most queries and you want to minimize strong model calls without pre-classifying anything.
Strategy 3: Semantic Routing
Semantic routing embeds the incoming query using a lightweight embedding model and matches it to the closest topic cluster. Each cluster maps to a model optimised for that domain.
Code queries go to a code-specialist model. Medical queries go to a model fine-tuned on clinical data. General conversation goes to the cheapest capable option.
This is different from complexity routing. You are not asking "how hard is this?" You are asking "what kind of task is this?" A simple medical question and a complex medical question both route to the medical model. A complex Python question and a simple Python question both route to the code model.
LiteLLM's auto routing supports this approach: it classifies each request using heuristics, a small LLM classifier (Haiku or GPT-4o mini), or lexical/semantic keyword rules, then routes to a pinned model or a scored pool of models per tier.
Best for: Teams running multiple specialised models or providers, or applications spanning clearly distinct task domains (support, code, analysis, creative).
Routing Strategies at a Glance
| Strategy | Routing Signal | Latency Overhead | Best For | Cost Reduction |
|---|---|---|---|---|
| Classifier-based | Predicted complexity score | Low (after training) | High-volume structured tasks | 45-85% |
| Cascade | Response confidence | Medium (model call overhead) | Mixed query types, no pre-classification needed | 50-98% |
| Semantic | Query topic embedding | Very low | Multi-domain, multi-model deployments | Varies by domain split |
)
Routing at the Gateway Layer
The most durable place to implement routing is not inside your application code. It is at the gateway layer.
When routing lives inside individual services, every team maintains its own logic. It drifts. It does not account for new models. Costs creep back up.
A gateway-level routing layer applies consistent rules to all LLM traffic across every application. You define the routing policy once. Every consumer gets it automatically. And you get the cost attribution data to see whether the routing is actually working.
NeuralTrust's AI Gateway provides routing policies alongside cost attribution, so you can see (per route, per consumer) how much of your traffic is hitting each model tier and what that costs. When combined with prompt compression and caching, gateway-level routing becomes the foundation for a complete token cost management system.
Open-Source Tools Worth Knowing
RouteLLM (UC Berkeley + Anyscale) is the most academically rigorous option. It ships with four router implementations: matrix factorization, BERT classifier, cosine similarity, and LLM-as-judge. You can use their pre-trained weights or train your own on your query distribution. The GitHub repo is open source. Best for teams who want strong theoretical guarantees and are comfortable with a training step.
LiteLLM is the most production-ready general-purpose option. It handles routing alongside load balancing, fallbacks, budget limits, and provider failover. Routing strategies include least-busy, latency-based, usage-based, and cost-based. The proxy setup means you do not change application code, you point everything at the LiteLLM proxy and configure routing in YAML. Best for teams who want routing plus full inference infrastructure in one layer.
Martian is a managed router with an ML-based approach that adapts to your traffic over time. It sits between your app and your LLM providers, learns which queries need which model, and optimises automatically. Best for teams who want routing without building or maintaining any infrastructure.
Each tool covers different parts of the same problem. RouteLLM for research-grade routing. LiteLLM for self-hosted production infrastructure. Martian for managed, zero-maintenance routing.
The Cost-Quality Trade-off in Practice
The routing decision is not binary, small or frontier. Most production systems benefit from three tiers.
| Query Type | Model Tier | Why |
|---|---|---|
| Classification, yes/no, structured extraction | Nano (GPT-4o mini, Gemini Flash) | Accuracy near-identical to frontier on structured tasks |
| Summarisation, moderate Q&A, translation | Mid-tier (Haiku, Gemini 1.5 Flash) | Strong at compression and retrieval; cheap |
| Complex reasoning, code generation, analysis | Frontier (GPT-4o, Claude Sonnet) | Required for multi-step and creative tasks |
| Domain-specific (medical, legal, code) | Specialised fine-tuned or RAG | Outperforms generalist on narrow tasks |
For a realistic production mix (60% simple, 30% moderate, 10% complex) routing to three tiers produces roughly 70-80% cost reduction versus routing everything to the frontier.
For more on managing context costs within each tier, see the Context Window Optimization guide. For caching repeated inputs before they even reach the router, see LLM Caching Strategies.
Frequently Asked Questions about LLM Model Routing
1. What is LLM model routing?
LLM model routing is the automatic process of directing each query to the most cost-effective model capable of handling it. A routing layer evaluates the incoming query (by complexity, topic, confidence, or other signals) and selects the appropriate model from a pool. This replaces the default behaviour of sending everything to a single, usually expensive, model.
2. How much can routing reduce LLM costs?
Research benchmarks put the range at 40-98% depending on query distribution and routing method. RouteLLM achieved 85% cost reduction on MT Bench at 95% of GPT-4 quality. FrugalGPT achieved up to 98% cost reduction across benchmarks. Real production savings depend on how skewed your query distribution is toward simple tasks.
3. What is the difference between cascade routing and classifier-based routing?
Classifier-based routing makes the routing decision before the query reaches any model, using a lightweight classifier trained to predict complexity. Cascade routing sends the query to the cheap model first and only escalates if the response does not meet a confidence threshold. Classifiers add no model call overhead; cascades add the overhead of the cheap model call on every query, including ones that would have been routed to the frontier anyway.
4. Does routing hurt output quality?
On tasks where small models are genuinely capable (structured extraction, classification, summarisation, yes/no questions) quality loss is negligible. On complex reasoning, open-ended generation, and multi-step tasks, routing to a weaker model will hurt quality. The key is calibrating which tasks actually need the frontier model. Most production workloads have a smaller percentage of genuinely hard queries than teams assume.
5. Which open-source routing tool should I use?
RouteLLM for teams who want rigorous, research-grade routing with pre-trained weights. LiteLLM for production infrastructure that combines routing with load balancing, fallbacks, and budget management in one proxy layer. Martian for a managed, zero-maintenance option that adapts to your traffic automatically.
6. What is semantic routing?
Semantic routing embeds the incoming query and matches it to a topic cluster, then routes to the model best suited for that domain. Unlike complexity routing (small vs. large model), semantic routing asks "what kind of task is this?" rather than "how hard is this task?". It is most useful when you run specialised models for distinct domains like code, medical, legal, or customer support.
Related articles
- AI Token Optimization: The Complete Enterprise Guide to Reducing LLM Costs (2026)
- Prompt Compression: How to Cut Token Costs Without Losing Output Quality
- LLM Caching Strategies: Prompt Caching, Semantic Caching, and When to Use Each
- LLM Cost Reduction: 12 Proven Strategies to Cut Your AI Inference Bill
- Context Window Optimization: How to Manage Long Contexts Without Blowing Your Budget
About the Author
Roger Howroyd is Head of Global SEO and AI at NeuralTrust, where he leads the company's search strategy across SEO, AEO, GEO, and LLM optimization. He specializes in AI-powered search, content strategy, backlink development, and SEM. Connect on LinkedIn
NeuralTrust is an AI agent security platform, recognized in the Gartner 2025 Market Guide for AI Gateways and Guardian Agents, and the KuppingerCole 2025 Leadership Compass for Generative AI Defense. Headquartered in Barcelona with ISO 27001 certification.
)
)
)
)