What is LLM model routing?
Model routing sends each request to the model that fits it. How routers decide, what research shows about savings, and where they fail.
Model routing is choosing, for each request, which model answers it. Anthropic describes a routing workflow as one that classifies an input and directs it to a specialized followup task, and gives two uses: sending customer service queries to different prompts and tools, and sending easy questions to smaller, cost-efficient models while hard ones go to more capable models. The first use is about fit, the second is about cost.
Routing is not the same as an LLM gateway. A gateway is the service in front of your providers, and routing is one decision it can make. You can also write routing as a few lines of code in your own app.
Two reasons to route
Cost is the common one. The FrugalGPT authors note that LLM API fees can differ by two orders of magnitude, so an easy query sent to the biggest model costs far more than it needs to. Quality is the other. Anthropic says that without routing, optimizing for one kind of input can hurt performance on other inputs, so separate prompts per category can beat one prompt for everything.
How routers decide
| Approach | How it picks | Source |
|---|---|---|
| Rules or a classifier | Classify the input, by an LLM or “a more traditional classification model/algorithm,” then send each category down its own path | Anthropic’s routing workflow |
| Difficulty prediction | A router predicts query difficulty and picks a small or large model for a quality level you can adjust at test time | Hybrid LLM (Ding et al.) |
| Preference-trained router | A router trained on human preference data decides whether the stronger, more expensive model is needed | RouteLLM (Ong et al.) |
| Cascade | A system learns which combination of models to use for each query, as an LLM cascade | FrugalGPT (Chen, Zaharia and Zou) |
The RouteLLM team describes four router types: a similarity-weighted ranking router, a matrix factorization model, a BERT classifier and a causal LLM classifier.
What the research reports
These are authors’ results on benchmarks, from 2023 and 2024, and the models they used are old. Treat them as evidence that routing can work and measure the savings on your own traffic.
| Study | Reported result |
|---|---|
| FrugalGPT (May 2023) | Matches GPT-4 performance with up to 98% cost reduction, or improves accuracy over GPT-4 by 4% at the same cost |
| Hybrid LLM (April 2024) | Cuts calls to the large model by up to 40% with no drop in response quality |
| RouteLLM (July 2024) | Against GPT-4 only, cost reductions of over 85% on MT Bench, 45% on MMLU and 35% on GSM8K, while achieving 95% of GPT-4’s performance |
The RouteLLM post also says its routers carried over to a different model pair (Claude 3 Opus and Llama 3 8B) without retraining. Its main test routed between GPT-4 Turbo and Mixtral 8x7B.
Where routers fail
- Out-of-distribution traffic. The RouteLLM authors report that routers trained only on Chatbot Arena data performed near random on MMLU, which they attribute to most MMLU questions being out of distribution. A router tuned on one kind of traffic can misjudge another.
- Savings depend on the baseline. The same post compares against GPT-4 only in some results and a random router in others, so its percentages are not interchangeable. It also treats weaker-model calls as negligible in its cost figures.
- Classification has to be accurate. Anthropic says routing works where “classification can be handled accurately.” A wrong label sends a request to the wrong prompt or model.
- Old results, new models. Anthropic’s post dates from December 2024, and the research above used models from 2023 and 2024. Re-test with the models you run today.
Compare routers on a benchmark
RouterBench is a public way to compare routing systems. Its authors released an evaluation framework and a dataset of over 405k inference outcomes from representative LLMs. For your own app, the same idea applies at small scale: collect real requests, run each candidate model, score the outputs and compare cost and quality for every routing rule. See LLM evaluation for scoring methods and LLM cost optimization for the other levers, such as semantic caching.
Check a routing rule in HoneyHive
Treat a routing rule as a change to test. HoneyHive’s Comparing Experiments page covers side-by-side comparison of runs to spot metric deltas across versions, so one run can use the router and another the strong model alone. In production, session aggregations give automatically aggregated cost and token totals, which shows whether the savings held. For tracing practice see LLM observability.
FAQ
What is model routing?
Model routing means choosing, per request, which model answers it. Anthropic describes a routing workflow as one that classifies an input and directs it to a specialized followup task, such as an easy question to a smaller model and a hard one to a more capable model.
How does a router decide which model to use?
Either with rules and a classifier, or with a model trained to predict the outcome. Hybrid LLM's router estimates query difficulty, and RouteLLM's routers learn from human preference data whether a stronger model is needed.
How much does model routing save?
It depends on the task and the baseline. The RouteLLM authors report cost reductions of over 85% on MT Bench, 45% on MMLU and 35% on GSM8K against GPT-4 alone, while keeping 95% of its performance. Those are research benchmarks, so measure your own traffic.
Is model routing the same as an LLM gateway?
No. Routing is the decision about which model gets a request. An LLM gateway is the service that sits in front of providers and may include routing among other features such as keys, retries and logging.
What can go wrong with a router?
A router can send a hard query to a weak model. The RouteLLM authors found that routers trained only on Chatbot Arena data performed near random on MMLU, which they attribute to most MMLU questions being out of distribution.