mlsanityData to 7 Oct 2026

When Does Confidence-Based Cascade Deferral Suffice?

arXiv 2307.02764 ↗First posted 6 Jul 2023cs.LG stat.MLcited by 58

Not on the map. Why?

Abstract (arXiv)

Cascades are a classical strategy to enable inference cost to vary adaptively across samples, wherein a sequence of classifiers are invoked in turn. A deferral rule determines whether to invoke the next classifier in the sequence, or to terminate prediction. One simple deferral rule employs the confidence of the current classifier, e.g., based on the maximum predicted softmax probability. Despite being oblivious to the structure of the cascade -- e.g., not modelling the errors of downstream models -- such confidence-based deferral often works remarkably well in practice. In this paper, we seek to better understand the conditions under which confidence-based deferral may fail, and when alternate deferral strategies can perform better. We first present a theoretical characterisation of the optimal deferral rule, which precisely characterises settings under which confidence-based deferral may suffer. We then study post-hoc deferral mechanisms, and demonstrate they can significantly improve upon confidence-based deferral in settings where (i) downstream models are specialists that only work well on a subset of inputs, (ii) samples are subject to label noise, and (iii) there is distribution shift between the train and test set.

Key facts. Cited by 58 papers in the map (data through 7 Oct 2026). Builds on 46 of its 86 references that are in the map. First posted on arXiv 6 Jul 2023.

Builds on

46 of 86 references.

References · 46

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  7. 2017On Calibration of Modern Neural Networkscited by 4,860
  8. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  10. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  11. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
  12. 2016Deep Networks with Stochastic Depthcited by 973
34 more
  1. 2019Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methodscited by 903
  2. 2014Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Imagescited by 774
  3. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
  4. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  5. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 413
  6. 2021Dynamic Neural Networks: A Surveycited by 405
  7. 2019SelectiveNet: A Deep Neural Network with an Integrated Reject Optioncited by 333
  8. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 322
  9. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 267
  10. 2020Consistent Estimators for Learning to Defer to an Expertcited by 262
  11. 2017Predict Responsibly: Improving Fairness and Accuracy by Learning to Defercited by 228
  12. 2019Depth-Adaptive Transformercited by 227
  13. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 224
  14. 2016FractalNet: Ultra-Deep Neural Networks without Residualscited by 216
  15. 2018To Trust Or Not To Trust A Classifiercited by 205
  16. 2020The Right Tool for the Job: Matching Model and Instance Complexitiescited by 125
  17. 2016An Analysis of Deep Neural Network Models for Practical Applicationscited by 114
  18. 2021Machine Learning with a Reject Option: A surveycited by 107
  19. 2019Combating Label Noise in Deep Learning Using Abstentioncited by 86
  20. 2020Why should we add early exits to neural networks?cited by 70
  21. 2015Conditional Deep Learning for Energy-Efficient and Enhanced Pattern Recognitioncited by 67
  22. 2020Classification with Rejection Based on Cost-sensitive Classificationcited by 55
  23. 2017IDK Cascades: Fast Deep Learning by Learning not to Overthinkcited by 47
  24. 2022Sample Efficient Learning of Predictors that Complement Humanscited by 46
  25. 2022Model Cascading: Towards Jointly Improving Efficiency and Accuracy of NLP Systemscited by 43
  26. 2019On the Calibration of Multiclass Classification with Rejectioncited by 41
  27. 2020Wisdom of Committees: An Overlooked Approach To Faster and More Accurate Modelscited by 39
  28. 2020Selective Classification via One-Sided Predictioncited by 37
  29. 2022TangoBERT: Reducing Inference Cost by using Cascaded Architecturecited by 13
  30. 2022BabyBear: Cheap inference triage for expensive language modelscited by 10
  31. 2018Approximation Algorithms for Cascading Prediction Modelscited by 9
  32. 2021When in Doubt, Summon the Titans: Efficient Inference with Large Modelscited by 8
  33. 2022Predicting on the Edge: Identifying Where a Larger Model Does Bettercited by 6
  34. 2020Any-Width Networkscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 13; 2025: 19; 2026: 25.

58 papers.

Sort:

By area:

  1. 2023Think before you speak: Training Language Models With Pause Tokenscited by 254
  2. 2024Language Model Cascades: Token-level uncertainty and beyondcited by 115
  3. 2025Harnessing Multiple Large Language Models: A Survey on LLM Ensemblecited by 86
  4. 2025Universal Model Routing for Efficient LLM Inferencecited by 80
  5. 2024A Unified Approach to Routing and Cascading for LLMscited by 71
  6. 2025Detecting High-Stakes Interactions with Activation Probescited by 45
  7. 2024Faster Cascades via Speculative Decodingcited by 38
  8. 2024Realizable H-Consistent and Bayes-Consistent Loss Functions for Learning to Defercited by 26
  9. 2024Online Cascade Learning for Efficient Inference over Streamscited by 19
  10. 2024Agreement-Based Cascading for Efficient Inferencecited by 16
  11. 2025Mastering Multiple-Expert Routing: Realizable H-Consistency and Strong Guarantees for Learning to Defercited by 16
  12. 2024Cascade-Aware Training of Language Modelscited by 15
38 more
  1. 2024Optimal Query Allocation in Extractive QA with LLMs: A Learning-to-Defer Framework with Theoretical Guaranteescited by 15
  2. 2025Cost-Saving LLM Cascades with Early Abstentioncited by 14
  3. 2025Budgeted Multiple-Expert Deferralcited by 11
  4. 2025Gatekeeper: Improving Model Cascades Through Confidence Tuningcited by 10
  5. 2026When Does Combining Language Models Help? A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents Across 67 Frontier Modelscited by 10
  6. 2025Rational Tuning of LLM Cascades via Probabilistic Modelingcited by 9
  7. 2024Privacy-preserved LLM Cascade via CoT-enhanced Policy Learningcited by 8
  8. 2025Why Ask One When You Can Ask k? Learning-to-Defer to the Top-k Expertscited by 6
  9. 2025ThinkingViT: Matryoshka Thinking Vision Transformer for Elastic Inferencecited by 6
  10. 2025Translate Smart, not Hard: Cascaded Translation Systems with Quality-Aware Deferralcited by 4
  11. 2025Cost-Aware Routing for Efficient Text-To-Image Generationcited by 4
  12. 2025Economic Evaluation of LLMscited by 4
  13. 2024Rejection via Learning Density Ratioscited by 3
  14. 2026Is Escalation Worth It? On the Depth of LLM Cascadescited by 3
  15. 2026Optimizing the Cost-Quality Tradeoff of Agentic Theorem Provers in Leancited by 3
  16. 2024EERO: Early Exit with Reject Option for Efficient Classification with limited budgetcited by 2
  17. 2024SMART: Automatically Scaling Down Language Models with Accuracy Guarantees for Reduced Processing Feescited by 2
  18. 2025Bi-directional Model Cascading with Proxy Confidencecited by 2
  19. 2025Inference-Time Distillation: Cost-Efficient Agents Without Fine-Tuning or Manual Prompt Engineeringcited by 2
  20. 2026When More Experts Hurt: Underfitting in Multi-Expert Learning to Defercited by 2
  21. 2026Ensembling Language Models with Sequential Monte Carlocited by 2
  22. 2026Streaming Model Cascades for Semantic SQLcited by 2
  23. 2024Predicting Probabilities of Error to Combine Quantization and Early Exiting: QuEEcited by 1
  24. 2025From Deferral to Learning: Online In-Context Knowledge Distillation for LLM Cascadescited by 1
  25. 2025Reliable Inference in Edge-Cloud Model Cascades via Conformal Alignmentcited by 1
  26. 2025Kad: A Framework for Proxy-based Test-time Alignment with Knapsack Approximation Deferralcited by 1
  27. 2026Multi-LLM Query Optimizationcited by 1
  28. 2026Density-Ratio Losses for Post-Hoc Learning to Defercited by 1
  29. 2026Knowing When to Ask for Help: Bayesian Self-Escalation in Hierarchical LLM Agentscited by 1
  30. 2026Return or Revise? Learning When Revision Helps Retrieval-Augmented QAcited by 1
  31. 2025Rethinking Calibration for Early-Exit Neural Networkscited by 0
  32. 2026LaCy: What Small Language Models Can and Should Learn is Not Just a Question of Losscited by 0
  33. 2026AdaDINO: Context-Adaptive DINO-Distilled Vision Foundation Models for Efficient Open-Vocabulary Edge Inferencecited by 0
  34. 2026A Regime Theory of Controller Class Selection for LLM Action Decisionscited by 0
  35. 2026AutoRelAnnotator: Calibrated Model Cascades for Cost-Efficient Relevance Evaluation in Sponsored Searchcited by 0
  36. 2026Expected Gain-based Escalation in Vertical Federated Learningcited by 0
  37. 2026Detect Early, Escalate Rarely: Anytime Detection of AI-Generated Video from the Compressed Bitstreamcited by 0
  38. 2026Conformal Cascade: Distribution-Free Accuracy Guarantees for Multi-Tier LLM Inferencecited by 0

50 most cited of 58 All 58 (JSON)