# Is Escalation Worth It? On the Depth of LLM Cascades

arXiv: https://arxiv.org/abs/2605.06350 · HTML: https://mlsanity.com/paper/2605.06350/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 30 references that are in the map.
- First posted on arXiv 7 May 2026.

## Builds on (most cited first, 28 of 28)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance](https://mlsanity.com/paper/2305.05176/) (2023)
- [Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback](https://mlsanity.com/paper/2305.14975/) (2023)
- [SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models](https://mlsanity.com/paper/2303.08896/) (2023)
- [RouteLLM: Learning to Route LLMs with Preference Data](https://mlsanity.com/paper/2406.18665/) (2024)
- [Measuring short-form factuality in large language models](https://mlsanity.com/paper/2411.04368/) (2024)
- [GraphRouter: A Graph-based Router for LLM Selections](https://mlsanity.com/paper/2410.03834/) (2024)
- [Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoning](https://mlsanity.com/paper/2310.03094/) (2023)
- [AutoMix: Automatically Mixing Language Models](https://mlsanity.com/paper/2310.12963/) (2023)
- [Language Model Cascades: Token-level uncertainty and beyond](https://mlsanity.com/paper/2404.10136/) (2024)
- [A Unified Approach to Routing and Cascading for LLMs](https://mlsanity.com/paper/2410.10347/) (2024)
- [BEST-Route: Adaptive LLM Routing with Test-Time Optimal Compute](https://mlsanity.com/paper/2506.22716/) (2025)
- [When Does Confidence-Based Cascade Deferral Suffice?](https://mlsanity.com/paper/2307.02764/) (2023)
- [OmniRouter: Budget and Performance Controllable Multi-LLM Routing](https://mlsanity.com/paper/2502.20576/) (2025)
- [CP-Router: An Uncertainty-Aware Router Between LLM and LRM](https://mlsanity.com/paper/2505.19970/) (2025)
- [Uncertainty Quantification for Language Models: A Suite of Black-Box, White-Box, LLM Judge, and Ensemble Scorers](https://mlsanity.com/paper/2504.19254/) (2025)
- [Cost-Saving LLM Cascades with Early Abstention](https://mlsanity.com/paper/2502.09054/) (2025)
- [When to Reason: Semantic Router for vLLM](https://mlsanity.com/paper/2510.08731/) (2025)
- [Efficient Contextual LLM Cascades through Budget-Constrained Policy Learning](https://mlsanity.com/paper/2404.13082/) (2024)
- [Rational Tuning of LLM Cascades via Probabilistic Modeling](https://mlsanity.com/paper/2501.09345/) (2025)
- [EAGer: Entropy-Aware GEneRation for Adaptive Inference-Time Scaling](https://mlsanity.com/paper/2510.11170/) (2025)
- [UQLM: A Python Package for Uncertainty Quantification in Large Language Models](https://mlsanity.com/paper/2507.06196/) (2025)
- [Privacy-preserved LLM Cascade via CoT-enhanced Policy Learning](https://mlsanity.com/paper/2410.08014/) (2024)
- [C3PO: Optimized Large Language Model Cascades with Probabilistic Cost Constraints for Reasoning](https://mlsanity.com/paper/2511.07396/) (2025)
- [Robust Batch-Level Query Routing for Large Language Models under Cost and Capacity Constraints](https://mlsanity.com/paper/2603.26796/) (2026)

## Built on by (most cited first, 4 of 4)

- [Return or Revise? Learning When Revision Helps Retrieval-Augmented QA](https://mlsanity.com/paper/2609.30087/) (2026)
- [Do Not Restart: Residual Completion for Stateful Agent Handoffs](https://mlsanity.com/paper/2609.13800/) (2026)
- [An Architecture for Long-Horizon Agents: Levels, Ticks and Cascaded Intelligence](https://mlsanity.com/paper/2609.19519/) (2026)
- [CreateScore: Domain-Theory-Informed Bayesian Routing for LLM-Based CV Screening](https://mlsanity.com/paper/2610.02972/) (2026)

## Abstract (arXiv)

LLM cascades, in which a cheap model defers to an expensive one on low-confidence queries, are widely used to reduce inference cost. Given a pool of models, a practitioner must decide how many models to include and where to set each deferral threshold. We derive first-order optimality conditions showing that, at an optimum, the ratio of expected accuracy gain to expected downstream cost is equal across deferral boundaries. A local search based on these conditions closely matches exhaustive search. We also derive an identity that decomposes the accuracy gain of score-based escalation over random escalation into two AUROC terms. Across five benchmarks and nine deferral scores, with model sequences and thresholds optimized from a pool of eight models, two-model cascades improve mean test-set accuracy over single-model selection by 2.1 to 8.2 percentage points. However, allowing more than two models does not improve mean test-set accuracy in 118 of 135 comparisons across scorers, datasets, and depth caps, and adds at most 0.43 percentage points. To understand the role of deferral scores in depth gains, we conduct counterfactual experiments with simulated confidence scores. When these scores have high AUROC and reflect only whether the current model answered correctly, allowing more than two models improves test-set accuracy on four of five benchmarks. However, these gains do not persist when the scores also reflect query difficulty shared across models, even at the same AUROC. These results suggest that gains from additional depth depend on how well the confidence score separates correct from incorrect answers for the current model compared with later models.