# Improving Predictor Reliability with Selective Recalibration

arXiv: https://arxiv.org/abs/2410.05407 · HTML: https://mlsanity.com/paper/2410.05407/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 39 references that are in the map.
- First posted on arXiv 7 Oct 2024.

## Builds on (most cited first, 23 of 23)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty](https://mlsanity.com/paper/1912.02781/) (2019)
- [Selective Classification for Deep Neural Networks](https://mlsanity.com/paper/1705.08500/) (2017)
- [Revisiting the Calibration of Modern Neural Networks](https://mlsanity.com/paper/2106.07998/) (2021)
- [SelectiveNet: A Deep Neural Network with an Integrated Reject Option](https://mlsanity.com/paper/1901.09192/) (2019)
- [Verified Uncertainty Calibration](https://mlsanity.com/paper/1909.10155/) (2019)
- [Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners](https://mlsanity.com/paper/2307.01928/) (2023)
- [On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks](https://mlsanity.com/paper/1905.11001/) (2019)
- [Predict Responsibly: Improving Fairness and Accuracy by Learning to Defer](https://mlsanity.com/paper/1711.06664/) (2017)
- [Mix-n-Match: Ensemble and Compositional Methods for Uncertainty Calibration in Deep Learning](https://mlsanity.com/paper/2003.07329/) (2020)
- [Mitigating Bias in Calibration Error Estimation](https://mlsanity.com/paper/2012.08668/) (2020)
- [Soft Calibration Objectives for Neural Networks](https://mlsanity.com/paper/2108.00106/) (2021)
- [When and How Mixup Improves Calibration](https://mlsanity.com/paper/2102.06289/) (2021)
- [Selective Classification Can Magnify Disparities Across Groups](https://mlsanity.com/paper/2010.14134/) (2020)
- [Selective Regression Under Fairness Criteria](https://mlsanity.com/paper/2110.15403/) (2021)
- [Calibrated Selective Classification](https://mlsanity.com/paper/2208.12084/) (2022)
- [Beyond calibration: estimating the grouping loss of modern neural networks](https://mlsanity.com/paper/2210.16315/) (2022)
- [Prompt Risk Control: A Rigorous Framework for Responsible Deployment of Large Language Models](https://mlsanity.com/paper/2311.13628/) (2023)
- [Quantile Risk Control: A Flexible Framework for Bounding the Probability of High-Loss Predictions](https://mlsanity.com/paper/2212.13629/) (2022)
- [Distribution-Free Statistical Dispersion Control for Societal Applications](https://mlsanity.com/paper/2309.13786/) (2023)

## Built on by (most cited first, 3 of 3)

- [Confidence Calibration in Vision-Language-Action Models](https://mlsanity.com/paper/2507.17383/) (2025)
- [Unsupervised Confidence Calibration for Reasoning LLMs from a Single Generation](https://mlsanity.com/paper/2604.19444/) (2026)
- [VLAConf: Calibrated Task-Success Confidence for Vision-Language-Action Models](https://mlsanity.com/paper/2605.29605/) (2026)

## Abstract (arXiv)

A reliable deep learning system should be able to accurately express its confidence with respect to its predictions, a quality known as calibration. One of the most effective ways to produce reliable confidence estimates with a pre-trained model is by applying a post-hoc recalibration method. Popular recalibration methods like temperature scaling are typically fit on a small amount of data and work in the model's output space, as opposed to the more expressive feature embedding space, and thus usually have only one or a handful of parameters. However, the target distribution to which they are applied is often complex and difficult to fit well with such a function. To this end we propose textitselective recalibration, where a selection model learns to reject some user-chosen proportion of the data in order to allow the recalibrator to focus on regions of the input space that can be well-captured by such a model. We provide theoretical analysis to motivate our algorithm, and test our method through comprehensive experiments on difficult medical imaging and zero-shot classification tasks. Our results show that selective recalibration consistently leads to significantly lower calibration error than a wide range of selection and recalibration baselines.