# Cost-Saving LLM Cascades with Early Abstention

arXiv: https://arxiv.org/abs/2502.09054 · HTML: https://mlsanity.com/paper/2502.09054/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 20 references that are in the map.
- First posted on arXiv 13 Feb 2025.

## Builds on (most cited first, 11 of 11)

- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance](https://mlsanity.com/paper/2305.05176/) (2023)
- [Selective Classification for Deep Neural Networks](https://mlsanity.com/paper/1705.08500/) (2017)
- [SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models](https://mlsanity.com/paper/2303.08896/) (2023)
- [Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models](https://mlsanity.com/paper/2305.19187/) (2023)
- [Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoning](https://mlsanity.com/paper/2310.03094/) (2023)
- [AutoMix: Automatically Mixing Language Models](https://mlsanity.com/paper/2310.12963/) (2023)
- [When Does Confidence-Based Cascade Deferral Suffice?](https://mlsanity.com/paper/2307.02764/) (2023)
- [Rational Tuning of LLM Cascades via Probabilistic Modeling](https://mlsanity.com/paper/2501.09345/) (2025)

## Built on by (most cited first, 14 of 14)

- [Conformal Arbitrage: Risk-Controlled Balancing of Competing Objectives in Language Models](https://mlsanity.com/paper/2506.00911/) (2025)
- [Cascaded Language Models for Cost-effective Human-AI Decision-Making](https://mlsanity.com/paper/2506.11887/) (2025)
- [BAS: A Decision-Theoretic Approach to Evaluating Large Language Model Confidence](https://mlsanity.com/paper/2604.03216/) (2026)
- [BAMAS: Structuring Budget-Aware Multi-Agent Systems](https://mlsanity.com/paper/2511.21572/) (2025)
- [Economic Evaluation of LLMs](https://mlsanity.com/paper/2507.03834/) (2025)
- [DiSRouter: Distributed Self-Routing for LLM Selections](https://mlsanity.com/paper/2510.19208/) (2025)
- [Is Escalation Worth It? On the Depth of LLM Cascades](https://mlsanity.com/paper/2605.06350/) (2026)
- [UCCI: Calibrated Uncertainty for Cost-Optimal LLM Cascade Routing](https://mlsanity.com/paper/2605.18796/) (2026)
- [Doomed from the Start: Early Abort of LLM Agent Episodes via a Recall-Controlled Probe Cascade](https://mlsanity.com/paper/2607.06503/) (2026)
- [From Deferral to Learning: Online In-Context Knowledge Distillation for LLM Cascades](https://mlsanity.com/paper/2509.22984/) (2025)
- [Strategic Decision Support for AI Agents](https://mlsanity.com/paper/2606.12587/) (2026)
- [Improving LLM Reliability through Hybrid Abstention and Adaptive Detection](https://mlsanity.com/paper/2602.15391/) (2026)
- [Auditing and Repairing LLM-as-Judge Failures in a Production Text-to-SQL Pipeline](https://mlsanity.com/paper/2609.30290/) (2026)
- [HARISSA: Inference-Time Self-Checks for Efficient and Safe Local Language Model Deployment](https://mlsanity.com/paper/2609.38006/) (2026)

## Abstract (arXiv)

LLM cascades deploy small LLMs to answer most queries, limiting the use of large and expensive LLMs to difficult queries. This approach can significantly reduce costs without impacting performance. However, risk-sensitive domains such as finance or medicine place an additional premium on avoiding model errors. Since even the most expensive models are susceptible to making mistakes, applications in these domains benefit from allowing LLM systems to completely abstain from answering difficult queries. Introducing abstention poses a design question for LLM cascades: should abstention only be allowed at the final model or also at earlier models? Since the error patterns of small and large models are correlated, allowing earlier models to abstain may reduce inference costs and latency by anticipating abstention decisions by expensive and slow models, thus avoiding the need to run these models. We investigate the benefits of such "early abstention" in LLM cascades and find that it reduces overall test loss by 2.2% on average across six benchmarks (GSM8K, MedMCQA, MMLU, TriviaQA, TruthfulQA, and XSum). These gains result from a more effective use of abstention, trading a 4.1% average increase in the overall abstention rate for a 13.0% reduction in cost and a 5.0% reduction in error rate. Our findings demonstrate the possibility of leveraging correlations between the error patterns of different language models to drive performance improvements for LLM systems with abstention.