mlsanityData to 7 Oct 2026

A Unified Approach to Routing and Cascading for LLMs

arXiv 2410.10347 ↗First posted 14 Oct 2024cs.CLcited by 71

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

The availability of a wide range of large language models (LLMs) embedded in various agentic systems has significantly increased the potential of model selection strategies to improve the cost-performance tradeoff. Existing strategies involve either routing, where a single model is chosen per query, or cascading, which sequentially runs increasingly larger models until a satisfactory answer is found. However, current approaches face three key limitations: they (1) lack formal proofs of optimality, (2) fail to identify the conditions under which these strategies are most effective to improve the cost-performance tradeoff, and (3) are unable to combine both paradigms for further improvements. To address these issues, we first derive a novel optimal strategy for cascading and prove the optimality of an existing routing strategy. Further, we propose cascade routing, a unified framework that integrates routing and cascading into a theoretically optimal strategy. Through our analysis, we identify good quality estimators as the critical factor for the success of model selection paradigms. Finally, in our experiments, we show that cascade routing consistently outperforms the individual approaches by a large margin and we analyze quality estimators to determine when routing and/or cascading are useful paradigms for model selection.

Key facts. Cited by 71 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 33 of its 44 references that are in the map. First posted on arXiv 14 Oct 2024.

Builds on

33 of 44 references.

References · 33

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  5. 2021Program Synthesis with Large Language Modelscited by 3,509
  6. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  7. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  8. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  9. 2024Qwen2.5-Coder Technical Reportcited by 1,151
  10. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  11. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  12. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 633
21 more
  1. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 354
  2. 2022Confident Adaptive Language Modelingcited by 297
  3. 2024RouterBench: A Benchmark for Multi-LLM Routing Systemcited by 222
  4. 2023Large Language Model Routing with Benchmark Datasetscited by 166
  5. 2023Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Modelscited by 141
  6. 2023AutoMix: Automatically Mixing Language Modelscited by 116
  7. 2024Language Model Cascades: Token-level uncertainty and beyondcited by 115
  8. 2023Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modelingcited by 94
  9. 2023Exploring the Benefits of Training Expert Language Models over Instruction Tuningcited by 80
  10. 2024MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixturescited by 73
  11. 2024Learning How Hard to Think: Input-Adaptive Allocation of LM Computationcited by 67
  12. 2023When Does Confidence-Based Cascade Deferral Suffice?cited by 58
  13. 2020CascadeBERT: Accelerating Inference of Pre-trained Language Models via Calibrated Complete Models Cascadecited by 47
  14. 2022Model Cascading: Towards Jointly Improving Efficiency and Accuracy of NLP Systemscited by 43
  15. 2024MetaLLM: A High-performant and Cost-efficient Dynamic Framework for Wrapping LLMscited by 39
  16. 2024Optimising Calls to Large Language Models with Uncertainty-Based Two-Tier Selectioncited by 32
  17. 2023Tryage: Real-time, intelligent Routing of User Prompts to Large Language Modelscited by 28
  18. 2024Online Cascade Learning for Efficient Inference over Streamscited by 19
  19. 2024OptLLM: Optimal Assignment of Queries to Large Language Modelscited by 14
  20. 2021Efficient Online ML API Selection for Multi-Label Classification Taskscited by 11
  21. 2022BabyBear: Cheap inference triage for expensive language modelscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 16; 2026: 55.

71 papers.

Sort:

By area:

  1. 2025Harnessing Multiple Large Language Models: A Survey on LLM Ensemblecited by 86
  2. 2025Universal Model Routing for Efficient LLM Inferencecited by 80
  3. 2025Doing More with Less: A Survey on Routing Strategies for Resource Optimisation in Large Language Model-Based Systemscited by 18
  4. 2025xRouter: Training Cost-Aware LLMs Orchestration System via Reinforcement Learningcited by 16
  5. 2026Dynamic Model Routing and Cascading for Efficient LLM Inference: A Surveycited by 16
  6. 2025Route-and-Reason: Scaling Large Language Model Reasoning with Reinforced Model Routercited by 13
  7. 2026EvolveRouter: Co-Evolving Routing and Prompt for Multi-Agent Question Answeringcited by 13
  8. 2026When Does Combining Language Models Help? A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents Across 67 Frontier Modelscited by 10
  9. 2025C3PO: Optimized Large Language Model Cascades with Probabilistic Cost Constraints for Reasoningcited by 8
  10. 2026Budget-Aware Agentic Routing via Boundary-Guided Trainingcited by 8
  11. 2025Cascadia: An Efficient Cascade Serving System for Large Language Modelscited by 5
  12. 2025ProxRouter: Proximity-Weighted LLM Query Routing for Improved Robustness to Outlierscited by 5
38 more
  1. 2025GMTRouter: Personalized LLM Router over Multi-turn User Interactionscited by 5
  2. 2025BAMAS: Structuring Budget-Aware Multi-Agent Systemscited by 5
  3. 2026Meissa: Multi-modal Medical Agentic Intelligencecited by 5
  4. 2025A note on conditional PAC-efficient reasoning in large language model routingcited by 3
  5. 2026HAPS: Hierarchical LLM Routing with Joint Architecture and Parameter Searchcited by 3
  6. 2026Federate the Router: Learning Language Model Routers with Sparse and Decentralized Evaluationscited by 3
  7. 2026Explainable Model Routing for Agentic Workflowscited by 3
  8. 2026CascadeDebate: Multi-Agent Deliberation for Cost-Aware LLM Cascadescited by 3
  9. 2026Calibrate-Then-Delegate: Safety Monitoring with Risk and Budget Guarantees via Model Cascadescited by 3
  10. 2026Is Escalation Worth It? On the Depth of LLM Cascadescited by 3
  11. 2026Optimizing the Cost-Quality Tradeoff of Agentic Theorem Provers in Leancited by 3
  12. 2026What Does a Routing Oracle Measure Under Stochastic Decoding? Coupling, Scorer Choice, and Single-Commit Ceilingscited by 3
  13. 2025LLM Chemistry Estimation for Multi-LLM Recommendationcited by 2
  14. 2025WebRouter: Query-specific Router via Variational Information Bottleneck for Cost-sensitive Web Agentcited by 2
  15. 2026Routing, Cascades, and User Choice for LLMscited by 2
  16. 2026From Inference Routing to Agent Orchestration: Declarative Policy Compilation with Cross-Layer Verificationcited by 2
  17. 2026RouteNLP: Closed-Loop LLM Routing with Conformal Cascading and Distillation Co-Optimizationcited by 2
  18. 2026R2V Agent: Teaching SLMs When to Ask for Helpcited by 2
  19. 2026Multi-Head Latent Control: A Unified Interface for LLM Agent Decision Makingcited by 2
  20. 2026CodeRescue: Budget-Calibrated Recovery Routing for Coding Agentscited by 2
  21. 2026WISERouter: LLM Routing with Workload Budget Constraintcited by 2
  22. 2026Uncertainty Is Not Enough: Value-of-Information Routing for Mixtures of LoRA Expertscited by 2
  23. 2026Anytime Safe PAC Efficient Reasoningcited by 1
  24. 2026HyPAC: Cost-Efficient LLMs-Human Hybrid Annotation with PAC Error Guaranteescited by 1
  25. 2026Multi-LLM Query Optimizationcited by 1
  26. 2026The Capability Frontier: Benchmarks Miss 82% of Model Performancecited by 1
  27. 2026Cache-Aware Prompt Compression:A Two-Tier Cost Model for LLM API Cachingcited by 1
  28. 2026MERA: Model Evolution and Routing with Skill Adaptation for Agentic Systems at Scalecited by 1
  29. 2026Error-Aware Reverse Auction Mechanism for Large Language Model Routingcited by 1
  30. 2026Dual-Layer Agentic Memory with Fast Write Routing and Slow Consolidationcited by 1
  31. 2025EMAFusion: A Self-Optimizing System for Seamless LLM Selection and Integrationcited by 0
  32. 2025T-TAMER: Provably Taming Trade-offs in ML Servingcited by 0
  33. 2025Two-Stage Voting for Robust and Efficient Suicide Risk Detection on Social Mediacited by 0
  34. 2026Knowledge Access Beats Model Size: Memory Augmented Routing for Persistent AI Agentscited by 0
  35. 2026RAGent: Physics-Aware Agentic Reasoning for Training-Free mmWave Human Activity Recognitioncited by 0
  36. 2026A Regime Theory of Controller Class Selection for LLM Action Decisionscited by 0
  37. 2026Flexible Routing via Uncertainty Decompositioncited by 0
  38. 2026LatentRouter: Can We Choose the Right Multimodal Model Before Seeing Its Answer?cited by 0

50 most cited of 71 All 71 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025LLM Bandit: Cost-Efficient LLM Generation via Preference-Conditioned Dynamic Routing
  2. 2025SATER: A Self-Aware and Token-Efficient Approach to Routing and Cascading
  3. 2026Cluster, Route, Escalate: Cascaded Framework for Cost-Aware LLM Serving
  4. 2025Adaptive LLM Routing under Budget Constraints
  5. 2026Agentic Routing: The Harness-Native Data Flywheel
  6. 2025MasRouter: Learning to Route LLMs for Multi-Agent Systems
  7. 2025Learning to Route LLMs from Bandit Feedback: One Policy, Many Trade-offs
  8. 2025DiSRouter: Distributed Self-Routing for LLM Selections