# Learning to Cascade: Confidence Calibration for Improving the Accuracy and Computational Cost of Cascade Inference Systems

arXiv: https://arxiv.org/abs/2104.09286 · HTML: https://mlsanity.com/paper/2104.09286/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 35 references that are in the map.
- First posted on arXiv 15 Apr 2021.

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](https://mlsanity.com/paper/1807.03888/) (2018)
- [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](https://mlsanity.com/paper/1706.02690/) (2017)
- [BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks](https://mlsanity.com/paper/1709.01686/) (2017)
- [Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data](https://mlsanity.com/paper/2002.11297/) (2020)
- [Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples](https://mlsanity.com/paper/1711.09325/) (2017)
- [Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration](https://mlsanity.com/paper/1910.12656/) (2019)
- [On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks](https://mlsanity.com/paper/1905.11001/) (2019)
- [Learning Confidence for Out-of-Distribution Detection in Neural Networks](https://mlsanity.com/paper/1802.04865/) (2018)
- [BottleNet++: An End-to-End Approach for Feature Compression in Device-Edge Co-Inference Systems](https://mlsanity.com/paper/1910.14315/) (2019)
- [Conditional Deep Learning for Energy-Efficient and Enhanced Pattern Recognition](https://mlsanity.com/paper/1509.08971/) (2015)
- [Communication-Computation Trade-Off in Resource-Constrained Edge Inference](https://mlsanity.com/paper/2006.02166/) (2020)
- [Robust Out-of-distribution Detection for Neural Networks](https://mlsanity.com/paper/2003.09711/) (2020)
- [Edge Intelligence: On-Demand Deep Learning Model Co-Inference with Device-Edge Synergy](https://mlsanity.com/paper/1806.07840/) (2018)
- [IDK Cascades: Fast Deep Learning by Learning not to Overthink](https://mlsanity.com/paper/1706.00885/) (2017)
- [Confidence Calibration for Convolutional Neural Networks Using Structured Dropout](https://mlsanity.com/paper/1906.09551/) (2019)
- [Network Offloading Policies for Cloud Robotics: a Learning-based Approach](https://mlsanity.com/paper/1902.05703/) (2019)
- [Adaptive Feeding: Achieving Fast and Accurate Detections by Adaptively Combining Object Detectors](https://mlsanity.com/paper/1707.06399/) (2017)
- [Is the Most Accurate AI the Best Teammate? Optimizing AI for Teamwork](https://mlsanity.com/paper/2004.13102/) (2020)

## Built on by (most cited first, 14 of 14)

- [Task-Oriented Communication for Multi-Device Cooperative Edge Inference](https://mlsanity.com/paper/2109.00172/) (2021)
- [What is the Role of Small Models in the LLM Era: A Survey](https://mlsanity.com/paper/2409.06857/) (2024)
- [Breaking the Ceiling of the LLM Community by Treating Token Generation as a Classification for Ensembling](https://mlsanity.com/paper/2406.12585/) (2024)
- [Agreement-Based Cascading for Efficient Inference](https://mlsanity.com/paper/2407.02348/) (2024)
- [Gatekeeper: Improving Model Cascades Through Confidence Tuning](https://mlsanity.com/paper/2502.19335/) (2025)
- [Turbo: Opportunistic Enhancement for Edge Video Analytics](https://mlsanity.com/paper/2207.00172/) (2022)
- [Finding the SWEET Spot: Analysis and Improvement of Adaptive Inference in Low Resource Settings](https://mlsanity.com/paper/2306.02307/) (2023)
- [Confidence Calibration for Intent Detection via Hyperspherical Space and Rebalanced Accuracy-Uncertainty Loss](https://mlsanity.com/paper/2203.09278/) (2022)
- [Hidden Heterogeneity: When to Choose Similarity-Based Calibration](https://mlsanity.com/paper/2202.01840/) (2022)
- [QuickNets: Saving Training and Preventing Overconfidence in Early-Exit Neural Architectures](https://mlsanity.com/paper/2212.12866/) (2022)
- [Perception and Semantic Aware Regularization for Sequential Confidence Calibration](https://mlsanity.com/paper/2305.19498/) (2023)
- [Cooperative Inference for Real-Time 3D Human Pose Estimation in Multi-Device Edge Networks](https://mlsanity.com/paper/2504.03052/) (2025)
- [CLCNet: Rethinking of Ensemble Modeling with Classification Confidence Network](https://mlsanity.com/paper/2205.09612/) (2022)
- [Flexible Routing via Uncertainty Decomposition](https://mlsanity.com/paper/2605.07805/) (2026)

## Abstract (arXiv)

Recently, deep neural networks have become to be used in a variety of applications. While the accuracy of deep neural networks is increasing, the confidence score, which indicates the reliability of the prediction results, is becoming more important. Deep neural networks are seen as highly accurate but known to be overconfident, making it important to calibrate the confidence score. Many studies have been conducted on confidence calibration. They calibrate the confidence score of the model to match its accuracy, but it is not clear whether these confidence scores can improve the performance of systems that use confidence scores. This paper focuses on cascade inference systems, one kind of systems using confidence scores, and discusses the desired confidence score to improve system performance in terms of inference accuracy and computational cost. Based on the discussion, we propose a new confidence calibration method, Learning to Cascade. Learning to Cascade is a simple but novel method that optimizes the loss term for confidence calibration simultaneously with the original loss term. Experiments are conducted using two datasets, CIFAR-100 and ImageNet, in two system settings, and show that naive application of existing calibration methods to cascade inference systems sometimes performs worse. However, Learning to Cascade always achieves a better trade-off between inference accuracy and computational cost. The simplicity of Learning to Cascade allows it to be easily applied to improve the performance of existing systems.