Route, don't upgrade: how model cascades cut enterprise AI spend
A ticket label and a merger analysis shouldn't cost the same per token. Routing sends each task to the smallest model that does it well.
The default enterprise AI architecture is one frontier model behind every feature. It is simple to build and expensive to run: a request to label a support ticket is billed at the same per-token rate as a request to analyze a merger agreement. As usage grows, so does a bill that is largely paying reasoning prices for routine work.
Audit the traffic first
Classify your AI requests by task type, and many workloads turn out to be dominated by classification, extraction, reformatting, short summarization, and retrieval-grounded lookup — tasks well within the reach of small, fast models. The frontier model's advantage concentrates in multi-step reasoning, ambiguous judgment, and genuinely novel problems.
Three patterns that work
- Routing. A lightweight classifier or rule set picks a model before the call. Ong et al. (2024) trained routers on human preference data in RouteLLM and reported reducing costs by more than half in some settings without compromising response quality.
- Cascades. Try the least expensive capable model first and escalate when a scorer judges the answer unreliable. In FrugalGPT, Chen, Zaharia and Zou (2023) reported matching the performance of the best individual model with up to 98% cost reduction on their benchmark tasks.
- Hybrid small–large inference. Ding et al. (2024) routed queries between a small and a large model by predicted difficulty, making up to 40% fewer calls to the large model with no drop in response quality — with the small model able to run on local hardware.
The goal isn't the cheapest model. It's the cheapest model that clears the quality bar for each task.
Quality is the constraint; evaluation is the control
Routing without evaluation is just a cost cut waiting to become an incident. Every task class needs a quality threshold measured on real examples. New routes should run in shadow mode alongside the current path, then roll out gradually, with automatic fallback to a stronger model when confidence is low.
Beyond routing
- Caching of exact and semantically similar requests.
- Prompt and context trimming, since input tokens are often the majority of spend.
- Batch processing for work that doesn't need an immediate answer.
- Task-specific small models trained for your highest-volume tasks — using frontier-model outputs only where provider terms permit it.
The same logic applies to coding agents
AI coding tools are the sharpest case of this pattern. Agents read whole files, scan repositories, and write predictable code, all on the same frontier model that handles the hard decisions. Those jobs are routine by any definition and easy to check, which makes them ideal candidates for a cheaper route.
References
- Chen, L., Zaharia, M. & Zou, J. (2023). FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance.
- Ong, I. et al. (2024). RouteLLM: Learning to Route LLMs with Preference Data.
- Ding, D. et al. (2024). Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing. ICLR.