# Calibration in Deep Learning: A Survey of the State-of-the-Art

arXiv: https://arxiv.org/abs/2308.01222 · HTML: https://mlsanity.com/paper/2308.01222/

## Key facts

- Cited by 75 papers in the map (data through 14 Oct 2026).
- Builds on 75 of its 133 references that are in the map.
- First posted on arXiv 2 Aug 2023.

## Builds on (most cited first, 30 of 75)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016)
- [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019)
- [Deep Anomaly Detection with Outlier Exposure](https://mlsanity.com/paper/1812.04606/) (2018)
- [A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification](https://mlsanity.com/paper/2107.07511/) (2021)
- [A Survey of Uncertainty in Deep Neural Networks](https://mlsanity.com/paper/2107.03342/) (2021)
- [Speech Recognition with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1303.5778/) (2013)
- [Network In Network](https://mlsanity.com/paper/1312.4400/) (2013)
- [A Simple Baseline for Bayesian Uncertainty in Deep Learning](https://mlsanity.com/paper/1902.02476/) (2019)
- [Variational Dropout and the Local Reparameterization Trick](https://mlsanity.com/paper/1506.02557/) (2015)
- [Regularizing Neural Networks by Penalizing Confident Output Distributions](https://mlsanity.com/paper/1701.06548/) (2017)
- [Measuring Calibration in Deep Learning](https://mlsanity.com/paper/1904.01685/) (2019)

## Built on by (most cited first, 30 of 75)

- [Approaching Human-Level Forecasting with Language Models](https://mlsanity.com/paper/2402.18563/) (2024)
- [Rewarding Doubt: A Reinforcement Learning Approach to Calibrated Confidence Expression of Large Language Models](https://mlsanity.com/paper/2503.02623/) (2025)
- [On the Challenges and Opportunities in Generative AI](https://mlsanity.com/paper/2403.00025/) (2024)
- [SWE-RM: Execution-free Feedback For Software Engineering Agents](https://mlsanity.com/paper/2512.21919/) (2025)
- [A Unified Perspective for Loss-Oriented Imbalanced Learning via Localization](https://mlsanity.com/paper/2310.04752/) (2023)
- [Reality Check: A New Evaluation Ecosystem Is Necessary to Understand AI's Real World Effects](https://mlsanity.com/paper/2505.18893/) (2025)
- [When is Multicalibration Post-Processing Necessary?](https://mlsanity.com/paper/2406.06487/) (2024)
- [DevBench: A multimodal developmental benchmark for language learning](https://mlsanity.com/paper/2406.10215/) (2024)
- [Does confidence calibration improve conformal prediction?](https://mlsanity.com/paper/2402.04344/) (2024)
- [Confidence Calibration of Classifiers with Many Classes](https://mlsanity.com/paper/2411.02988/) (2024)
- [Calibrating Bayesian Learning via Regularization, Confidence Minimization, and Selective Inference](https://mlsanity.com/paper/2404.11350/) (2024)
- [Solar Photovoltaic Assessment with Large Language Model](https://mlsanity.com/paper/2507.19144/) (2025)
- [Rethinking the Uncertainty: A Critical Review and Analysis in the Era of Large Language Models](https://mlsanity.com/paper/2410.20199/) (2024)
- [Confidence-calibrated covariate shift correction for few-shot classification in Vision-Language Models](https://mlsanity.com/paper/2502.07847/) (2025)
- [Improving the Finite Sample Estimation of Average Treatment Effects using Double/Debiased Machine Learning with Propensity Score Calibration](https://mlsanity.com/paper/2409.04874/) (2024)
- [Improving Calibration by Relating Focal Loss, Temperature Scaling, and Properness](https://mlsanity.com/paper/2408.11598/) (2024)
- [Performance Control in Early Exiting to Deploy Large Models at the Same Cost of Smaller Ones](https://mlsanity.com/paper/2412.19325/) (2024)
- [Calibeating Prediction-Powered Inference](https://mlsanity.com/paper/2604.21260/) (2026)
- [Multimodal Graph Representation Learning for Robust Surgical Workflow Recognition with Adversarial Feature Disentanglement](https://mlsanity.com/paper/2505.01766/) (2025)
- [Improving Perturbation-based Explanations by Understanding the Role of Uncertainty Calibration](https://mlsanity.com/paper/2511.10439/) (2025)
- [Achieving Well-Informed Decision-Making in Drug Discovery: A Comprehensive Calibration Study using Neural Network-Based Structure-Activity Models](https://mlsanity.com/paper/2407.14185/) (2024)
- [SIDDA: SInkhorn Dynamic Domain Adaptation for Image Classification with Equivariant Neural Networks](https://mlsanity.com/paper/2501.14048/) (2025)
- [Technical report on label-informed logit redistribution for better domain generalization in low-shot classification with foundation models](https://mlsanity.com/paper/2501.17595/) (2025)
- [DEUCE: Dual-diversity Enhancement and Uncertainty-awareness for Cold-start Active Learning](https://mlsanity.com/paper/2502.00305/) (2025)
- [Pre-trained Models Succeed in Medical Imaging with Representation Similarity Degradation](https://mlsanity.com/paper/2503.07958/) (2025)
- [Towards Understanding The Calibration Benefits of Sharpness-Aware Minimization](https://mlsanity.com/paper/2505.23866/) (2025)
- [Knowing When to Quit: Probabilistic Early Exits for Speech Separation](https://mlsanity.com/paper/2507.09768/) (2025)
- [Multiclass Local Calibration with the Jensen-Shannon Distance](https://mlsanity.com/paper/2510.26566/) (2025)
- [PyHealth 2.0: A Comprehensive Open-Source Toolkit for Accessible and Reproducible Clinical Deep Learning](https://mlsanity.com/paper/2601.16414/) (2026)
- [Bayesian-LoRA: Probabilistic Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2601.21003/) (2026)

## Abstract (arXiv)

Calibrating deep neural models plays an important role in building reliable, robust AI systems in safety-critical applications. Recent work has shown that modern neural networks that possess high predictive capability are poorly calibrated and produce unreliable model predictions. Though deep learning models achieve remarkable performance on various benchmarks, the study of model calibration and reliability is relatively under-explored. Ideal deep models should have not only high predictive performance but also be well calibrated. There have been some recent advances in calibrating deep models. In this survey, we review the state-of-the-art calibration methods and their principles for performing model calibration. First, we start with the definition of model calibration and explain the root causes of model miscalibration. Then we introduce the key metrics that can measure this aspect. It is followed by a summary of calibration methods that we roughly classify into four categories: post-hoc calibration, regularization methods, uncertainty estimation, and composition methods. We also cover recent advancements in calibrating large models, particularly large language models (LLMs). Finally, we discuss some open issues, challenges, and potential directions.