mlsanityData to 14 Oct 2026

Cluster, Route, Escalate: Cascaded Framework for Cost-Aware LLM Serving

arXiv 2606.27457 ↗First posted 25 Jun 2026cs.PF cs.CLcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Efficient deployment of large language models (LLMs) in production forces a trade-off between accuracy and cost. Operators often default to a single model that is either expensive for easy queries or insufficient for hard ones. To address this challenge, we propose a two-stage cascaded solution. Stage 1 clusters incoming queries and assigns each cluster to its most cost-effective model. The cost budget for this routing process is set by an interpretable hyperparameter, tuned offline. Stage 2 adds a quality estimation (QE) cascade; when an output from Stage 1 is judged low-quality, the query is escalated to a stronger model. This ensures only hard or low-confidence cases reach the expensive models. On the test datasets, the cascaded system retains 97-99% of the strongest model's accuracy while reducing Time Per Output Token (TPOT). It requires only task-correctness labels and adapts to changes in the model pool without manual reconfiguration.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 20 of its 29 references that are in the map. First posted on arXiv 25 Jun 2026.

Builds on

20 of 29 references.

Closest ancestors · 2

Same direction, newest first

  1. 2026Dynamic Model Routing and Cascading for Efficient LLM Inference: A Survey23 Feb
  2. 2025Universal Model Routing for Efficient LLM Inference12 Feb

Other references · 18

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  3. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  4. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  5. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
  6. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 682
  7. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 554
  8. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 372
  9. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 263
  10. 2025AdaptThink: Reasoning Models Can Learn When to Thinkcited by 147
  11. 2023AutoMix: Automatically Mixing Language Modelscited by 125
  12. 2024Learning to Route LLMs with Confidence Tokenscited by 67
6 more
  1. 2023TeleQnA: A Benchmark Dataset to Assess Large Language Models Telecommunications Knowledgecited by 63
  2. 2025Confident or Seek Stronger: Exploring Uncertainty-Based On-device LLM Routing From Benchmarking to Generalizationcited by 29
  3. 2025When to Continue Thinking: Adaptive Thinking Mode Switching for Efficient Reasoningcited by 26
  4. 2025Tiny Model, Big Logic: Diversity-Driven Optimization Elicits Large-Model Reasoning Ability in VibeThinker-1.5Bcited by 8
  5. 2025Translate Smart, not Hard: Cascaded Translation Systems with Quality-Aware Deferralcited by 4
  6. 2026TeleCom-Bench: How Far Are Large Language Models from Industrial Telecommunication Applications?cited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Cascadia: An Efficient Cascade Serving System for Large Language Models
  2. 2024A Unified Approach to Routing and Cascading for LLMs
  3. 2025From Large to Super-Tiny: End-to-End Optimization for Cost-Efficient LLMs
  4. 2026Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving
  5. 2024Efficiently Deploying LLMs with Controlled Risk
  6. 2024TensorOpera Router: A Multi-Model Router for Efficient LLM Inference
  7. 2026RouteNLP: Closed-Loop LLM Routing with Conformal Cascading and Distillation Co-Optimization
  8. 2025Doing More with Less: A Survey on Routing Strategies for Resource Optimisation in Large Language Model-Based Systems