# Improving model calibration with accuracy versus uncertainty optimization

arXiv: https://arxiv.org/abs/2012.07923 · HTML: https://mlsanity.com/paper/2012.07923/

## Key facts

- Cited by 123 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Dec 2020.

## Built on by (most cited first, 30 of 123)

- [Uncertainty Quantification with Pre-trained Language Models: A Large-Scale Empirical Analysis](https://mlsanity.com/paper/2210.04714/) (2022)
- [Plex: Towards Reliability using Pretrained Large Model Extensions](https://mlsanity.com/paper/2207.07411/) (2022)
- [Evidential Deep Learning for Open Set Action Recognition](https://mlsanity.com/paper/2107.10161/) (2021)
- [Soft Calibration Objectives for Neural Networks](https://mlsanity.com/paper/2108.00106/) (2021)
- [Calibration in Deep Learning: A Survey of the State-of-the-Art](https://mlsanity.com/paper/2308.01222/) (2023)
- [Self-Consistency Preference Optimization](https://mlsanity.com/paper/2411.04109/) (2024)
- [Dual Focal Loss for Calibration](https://mlsanity.com/paper/2305.13665/) (2023)
- [Meta-Calibration: Learning of Model Calibration Using Differentiable Expected Calibration Error](https://mlsanity.com/paper/2106.09613/) (2021)
- [Towards Improving Calibration in Object Detection Under Domain Shift](https://mlsanity.com/paper/2209.07601/) (2022)
- [Taxonomy of Machine Learning Safety: A Survey and Primer](https://mlsanity.com/paper/2106.04823/) (2021)
- [NP-Match: When Neural Processes meet Semi-Supervised Learning](https://mlsanity.com/paper/2207.01066/) (2022)
- [Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach](https://mlsanity.com/paper/2505.01997/) (2025)
- [COLD Fusion: Calibrated and Ordinal Latent Distribution Fusion for Uncertainty-Aware Multimodal Emotion Recognition](https://mlsanity.com/paper/2206.05833/) (2022)
- [Data-IQ: Characterizing subgroups with heterogeneous outcomes in tabular data](https://mlsanity.com/paper/2210.13043/) (2022)
- [Confidence and Dispersity Speak: Characterising Prediction Matrix for Unsupervised Accuracy Estimation](https://mlsanity.com/paper/2302.01094/) (2023)
- [Single Model Uncertainty Estimation via Stochastic Data Centering](https://mlsanity.com/paper/2207.07235/) (2022)
- [Measuring and Improving Model-Moderator Collaboration using Uncertainty Estimation](https://mlsanity.com/paper/2107.04212/) (2021)
- [Multiclass Confidence and Localization Calibration for Object Detection](https://mlsanity.com/paper/2306.08271/) (2023)
- [MeteoRA: Multiple-tasks Embedded LoRA for Large Language Models](https://mlsanity.com/paper/2405.13053/) (2024)
- [A Survey on Safe Multi-Modal Learning System](https://mlsanity.com/paper/2402.05355/) (2024)
- [ESD: Expected Squared Difference as a Tuning-Free Trainable Calibration Measure](https://mlsanity.com/paper/2303.02472/) (2023)
- [Rethinking Data Distillation: Do Not Overlook Calibration](https://mlsanity.com/paper/2307.12463/) (2023)
- [Bridging Precision and Confidence: A Train-Time Loss for Calibrating Object Detection](https://mlsanity.com/paper/2303.14404/) (2023)
- [A Benchmark Study on Calibration](https://mlsanity.com/paper/2308.11838/) (2023)
- [Cal-DETR: Calibrated Detection Transformer](https://mlsanity.com/paper/2311.03570/) (2023)
- [Beyond Probability Partitions: Calibrating Neural Networks with Semantic Aware Grouping](https://mlsanity.com/paper/2306.04985/) (2023)
- [A Hierarchical Spatial Transformer for Massive Point Samples in Continuous Space](https://mlsanity.com/paper/2311.04434/) (2023)
- [Dense Uncertainty Estimation](https://mlsanity.com/paper/2110.06427/) (2021)
- [Calibrating a Deep Neural Network with Its Predecessors](https://mlsanity.com/paper/2302.06245/) (2023)
- [Uncertainty-guided Boundary Learning for Imbalanced Social Event Detection](https://mlsanity.com/paper/2310.19247/) (2023)

## Abstract (arXiv)

Obtaining reliable and accurate quantification of uncertainty estimates from deep neural networks is important in safety-critical applications. A well-calibrated model should be accurate when it is certain about its prediction and indicate high uncertainty when it is likely to be inaccurate. Uncertainty calibration is a challenging problem as there is no ground truth available for uncertainty estimates. We propose an optimization method that leverages the relationship between accuracy and uncertainty as an anchor for uncertainty calibration. We introduce a differentiable accuracy versus uncertainty calibration (AvUC) loss function that allows a model to learn to provide well-calibrated uncertainties, in addition to improved accuracy. We also demonstrate the same methodology can be extended to post-hoc uncertainty calibration on pretrained models. We illustrate our approach with mean-field stochastic variational inference and compare with state-of-the-art methods. Extensive experiments demonstrate our approach yields better model calibration than existing methods on large-scale image classification tasks under distributional shift.