mlsanityData to 14 Oct 2026

Measuring Calibration in Deep Learning

arXiv 1904.01685 ↗First posted 2 Apr 2019cs.LG stat.MLcited by 446

Not on the map. Why?

Abstract (arXiv)

Overconfidence and underconfidence in machine learning classifiers is measured by calibration: the degree to which the probabilities predicted for each class match the accuracy of the classifier on that prediction. How one measures calibration remains a challenge: expected calibration error, the most popular metric, has numerous flaws which we outline, and there is no clear empirical understanding of how its choices affect conclusions in practice, and what recommendations there are to counteract its flaws. In this paper, we perform a comprehensive empirical study of choices in calibration measures including measuring all probabilities rather than just the maximum prediction, thresholding probability values, class conditionality, number of bins, bins that are adaptive to the datapoint density, and the norm used to compare accuracies to confidences. To analyze the sensitivity of calibration measures, we study the impact of optimizing directly for each variant with recalibration techniques. Across MNIST, Fashion MNIST, CIFAR-10/100, and ImageNet, we find that conclusions on the rank ordering of recalibration methods is drastically impacted by the choice of calibration measure. We find that conditioning on the class leads to more effective calibration evaluations, and that using the L2 norm rather than the L1 norm improves both optimization for calibration metrics and the rank correlation measuring metric consistency. Adaptive binning schemes lead to more stablity of metric rank ordering when the number of bins vary, and is also recommended. We open source a library for the use of our calibration measures.

Key facts. Cited by 446 papers in the map (data through 14 Oct 2026). First posted on arXiv 2 Apr 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 44; 2022: 65; 2023: 85; 2024: 86; 2025: 83; 2026: 83.

446 papers.

Sort:

By area:

  1. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  2. 2021A Survey of Uncertainty in Deep Neural Networkscited by 680
  3. 2021Revisiting the Calibration of Modern Neural Networkscited by 390
  4. 2023Generating with Confidence: Uncertainty Quantification for Black-box Large Language Modelscited by 293
  5. 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 256
  6. 2023Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolutioncited by 229
  7. 2023MiniLLM: On-Policy Distillation of Large Language Modelscited by 139
  8. 2023Document Understanding Dataset and Evaluation (DUDE)cited by 135
  9. 2021On Calibration and Out-of-domain Generalizationcited by 134
  10. 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 121
  11. 2021Assessing Generalization of SGD via Disagreementcited by 117
  12. 2023Calibrated Language Models Must Hallucinatecited by 115
38 more
  1. 2022Plex: Towards Reliability using Pretrained Large Model Extensionscited by 102
  2. 2021Soft Calibration Objectives for Neural Networkscited by 92
  3. 2021Label Leakage and Protection in Two-party Split Learningcited by 90
  4. 2023Provable Dynamic Fusion for Low-Quality Multimodal Datacited by 82
  5. 2023A Survey on Uncertainty Quantification Methods for Deep Learningcited by 81
  6. 2023Calibration in Deep Learning: A Survey of the State-of-the-Artcited by 75
  7. 2023On the Calibration of Large Language Models and Alignmentcited by 73
  8. 2021Prior and Posterior Networks: A Survey on Evidential Deep Learning Methods For Uncertainty Estimationcited by 71
  9. 2022Stop Measuring Calibration When Humans Disagreecited by 71
  10. 2024Exponential Moving Average of Weights in Deep Learning: Dynamics and Benefitscited by 71
  11. 2022Repair Is Nearly Generation: Multilingual Program Repair with LLMscited by 68
  12. 2022MEDFAIR: Benchmarking Fairness for Medical Imagingcited by 68
  13. 2022Better Uncertainty Calibration via Proper Scores for Classification and Beyondcited by 65
  14. 2025Reasoning Models Better Express Their Confidencecited by 65
  15. 2022Trustworthy clinical AI solutions: a unified review of uncertainty quantification in deep learning models for medical image analysiscited by 63
  16. 2023Smooth ECE: Principled Reliability Diagrams via Kernel Smoothingcited by 63
  17. 2021When and How Mixup Improves Calibrationcited by 60
  18. 2022A Close Look into the Calibration of Pre-trained Language Modelscited by 57
  19. 2022Assaying Out-Of-Distribution Generalization in Transfer Learningcited by 53
  20. 2022A Unifying Theory of Distance from Calibrationcited by 53
  21. 2022Improving the Fairness of Chest X-ray Classifierscited by 52
  22. 2022A Stitch in Time Saves Nine: A Train-Time Regularizing Loss for Improved Neural Network Calibrationcited by 49
  23. 2022A Simple Approach to Improve Single-Model Deep Uncertainty via Distance-Awarenesscited by 49
  24. 2024Calibration and Correctness of Language Models for Codecited by 49
  25. 2021Shallow Bayesian Meta Learning for Real-World Few-Shot Recognitioncited by 48
  26. 2021DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimationcited by 48
  27. 2021Parameterized Temperature Scaling for Boosting the Expressive Power in Post-Hoc Uncertainty Calibrationcited by 48
  28. 2022Sample-dependent Adaptive Temperature Scaling for Improved Calibrationcited by 48
  29. 2021Top-label calibration and multiclass-to-binary reductionscited by 46
  30. 2021Deep Cox Mixtures for Survival Regressioncited by 44
  31. 2023Rethinking Confidence Calibration for Failure Predictioncited by 43
  32. 2024Uncertainty in Language Models: Assessment through Rank-Calibrationcited by 41
  33. 2021Triggering Failures: Out-Of-Distribution detection by learning from local adversarial attacks in Semantic Segmentationcited by 40
  34. 2022Metrics for saliency map evaluation of deep learning explanation methodscited by 40
  35. 2021Don't Just Blame Over-parametrization for Over-confidence: Theoretical Analysis of Calibration in Binary Classificationcited by 39
  36. 2021Towards Calibrated Model for Long-Tailed Visual Recognition from Prior Perspectivecited by 39
  37. 2023What Can We Learn From The Selective Prediction And Uncertainty Estimation Performance Of 523 Imagenet Classifierscited by 35
  38. 2022Predictive Inference with Feature Conformal Predictioncited by 34

50 most cited of 446 All 446 (JSON)