# Optimizing Calibration by Gaining Aware of Prediction Correctness

arXiv: https://arxiv.org/abs/2404.13016 · HTML: https://mlsanity.com/paper/2404.13016/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 34 references that are in the map.
- First posted on arXiv 19 Apr 2024.

## Builds on (most cited first, 27 of 27)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [Learning Robust Global Representations by Penalizing Local Predictive Power](https://mlsanity.com/paper/1905.13549/) (2019)
- [Measuring Calibration in Deep Learning](https://mlsanity.com/paper/1904.01685/) (2019)
- [Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration](https://mlsanity.com/paper/1910.12656/) (2019)
- [Calibrating Deep Neural Networks using Focal Loss](https://mlsanity.com/paper/2002.09437/) (2020)
- [Verified Uncertainty Calibration](https://mlsanity.com/paper/1909.10155/) (2019)
- [Mix-n-Match: Ensemble and Compositional Methods for Uncertainty Calibration in Deep Learning](https://mlsanity.com/paper/2003.07329/) (2020)
- [Improving model calibration with accuracy versus uncertainty optimization](https://mlsanity.com/paper/2012.07923/) (2020)
- [Calibration of Neural Networks using Splines](https://mlsanity.com/paper/2006.12800/) (2020)
- [The iWildCam 2020 Competition Dataset](https://mlsanity.com/paper/2004.10340/) (2020)
- [Post-hoc Uncertainty Calibration for Domain Drift Scenarios](https://mlsanity.com/paper/2012.10988/) (2020)
- [Parameterized Temperature Scaling for Boosting the Expressive Power in Post-Hoc Uncertainty Calibration](https://mlsanity.com/paper/2102.12182/) (2021)
- [Sample-dependent Adaptive Temperature Scaling for Improved Calibration](https://mlsanity.com/paper/2207.06211/) (2022)
- [Transferable Calibration with Lower Bias and Variance in Domain Adaptation](https://mlsanity.com/paper/2007.08259/) (2020)
- [Confidence Estimation via Auxiliary Models](https://mlsanity.com/paper/2012.06508/) (2020)
- [Dual Focal Loss for Calibration](https://mlsanity.com/paper/2305.13665/) (2023)
- [Confidence Calibration for Domain Generalization under Covariate Shift](https://mlsanity.com/paper/2104.00742/) (2021)
- [Adaptive Temperature Scaling for Robust Calibration of Deep Neural Networks](https://mlsanity.com/paper/2208.00461/) (2022)
- [On the Strong Correlation Between Model Invariance and Generalization](https://mlsanity.com/paper/2207.07065/) (2022)
- [Proximity-Informed Calibration for Deep Neural Networks](https://mlsanity.com/paper/2306.04590/) (2023)
- [Window-Based Early-Exit Cascades for Uncertainty Estimation: When Deep Ensembles are More Efficient than Single Models](https://mlsanity.com/paper/2303.08010/) (2023)
- [Beyond In-Domain Scenarios: Robust Density-Aware Calibration](https://mlsanity.com/paper/2302.05118/) (2023)
- [Adaptive Multi-head Contrastive Learning](https://mlsanity.com/paper/2310.05615/) (2023)
- [Scaling of Class-wise Training Losses for Post-hoc Calibration](https://mlsanity.com/paper/2306.10989/) (2023)

## Built on by (most cited first, 4 of 4)

- [Confidence Calibration of Classifiers with Many Classes](https://mlsanity.com/paper/2411.02988/) (2024)
- [Sample Margin-Aware Recalibration of Temperature Scaling](https://mlsanity.com/paper/2506.23492/) (2025)
- [Revisiting Reweighted Risk for Calibration: AURC, Focal, and Inverse Focal Loss](https://mlsanity.com/paper/2505.23463/) (2025)
- [Towards Reliable Detection of Empty Space: Conditional Marked Point Processes for Object Detection](https://mlsanity.com/paper/2506.21486/) (2025)

## Abstract (arXiv)

Model calibration aims to align confidence with prediction correctness. The Cross-Entropy (CE) loss is widely used for calibrator training, which enforces the model to increase confidence on the ground truth class. However, we find the CE loss has intrinsic limitations. For example, for a narrow misclassification (e.g., a test sample is wrongly classified and its softmax score on the ground truth class is 0.4), a calibrator trained by the CE loss often produces high confidence on the wrongly predicted class, which is undesirable. In this paper, we propose a new post-hoc calibration objective derived from the aim of calibration. Intuitively, the proposed objective function asks that the calibrator decrease model confidence on wrongly predicted samples and increase confidence on correctly predicted samples. Because a sample itself has insufficient ability to indicate correctness, we use its transformed versions (e.g., rotated, greyscaled, and color-jittered) during calibrator training. Trained on an in-distribution validation set and tested with isolated, individual test samples, our method achieves competitive calibration performance on both in-distribution and out-of-distribution test sets compared with the state of the art. Further, our analysis points out the difference between our method and commonly used objectives such as CE loss and Mean Square Error (MSE) loss, where the latters sometimes deviates from the calibration aim.