Smart Model Routing for Order-of-Magnitude LLM Cost Reduction
Not on the map. Why?
Abstract (arXiv)
Frontier large language models (LLMs) deliver the strongest response quality but carry per-token prices often more than an order of magnitude above those of smaller, task-adequate models. Smart model routing addresses this mismatch by assigning each query to the cheapest model expected to answer it adequately, rather than sending every request to the most capable, most expensive endpoint. This paper develops a formal cost-quality optimization framework for routing -- including a derived optimal threshold-policy structure, a calibration-versus-discrimination distinction, a cascade breakeven condition, and a sensitivity analysis over price ratio and traffic skew -- situates that framework against the principal routing architectures and the signals used to estimate query difficulty, and works through a numerical example showing how a calibrated router can reduce blended inference cost by roughly an order of magnitude relative to an always-frontier deployment under specific, stated pricing and traffic assumptions, reporting both the gross reduction from routing policy alone and the net figure with the router's own overhead priced in. We situate this figure against the range reported by published routing systems (roughly 2x to frontier-level accuracy at one-fifth the cost), and close with the practical risks -- under- and over-routing, aggregate quality constraints masking segment-level harm, calibration without discrimination, calibration drift, prompt injection into routing signals, and operational failures such as tier outages and multi-provider price variation -- that determine whether the savings materialize in production. The analysis is primarily analytical, complemented by a GSM8K evaluation of the cascade framework (Section 5), validated (Section 5.2); extending the evaluation across tasks and model families is future work.
Key facts. No citing papers in the map yet (data through 12 Nov 2026). Builds on 8 of its 9 references that are in the map. First posted on arXiv 11 Nov 2026.
Builds on
8 of 9 references.
References · 8
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,714
- 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 965
- 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 776
- 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 405
- 2024RouterBench: A Benchmark for Multi-LLM Routing Systemcited by 271
- 2023Large Language Model Routing with Benchmark Datasetscited by 186
- 2025BEST-Route: Adaptive LLM Routing with Test-Time Optimal Computecited by 76
- 2025Doing More with Less: A Survey on Routing Strategies for Resource Optimisation in Large Language Model-Based Systemscited by 22
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026TokenRouter: Efficient Serving System for Token-Level LLM Routing
- 2025Efficient Training-Free Online Routing for High-Volume Multi-LLM Serving
- 2026Rethinking Quality in Large Language Model Routing: Security-Conditioned Model-and-Budget Selection
- 2026R2-Router: A New Paradigm for LLM Routing with Reasoning
- 2026Cluster, Route, Escalate: Cascaded Framework for Cost-Aware LLM Serving
- 2024TensorOpera Router: A Multi-Model Router for Efficient LLM Inference
- 2024Performance Characterization of Expert Router for Scalable LLM Inference
- 2026RouteNLP: Closed-Loop LLM Routing with Conformal Cascading and Distillation Co-Optimization