mlsanityData to 7 Oct 2026

Understanding Model Calibration -- A gentle introduction and visual exploration of calibration and the expected calibration error (ECE)

arXiv 2501.19047 ↗First posted 31 Jan 2025stat.ME cs.AI cs.CVcited by 20

Not on the map. Why?

Abstract (arXiv)

To be considered reliable, a model must be calibrated so that its confidence in each decision closely reflects its true outcome. In this blogpost we'll take a look at the most commonly used definition for calibration and then dive into a frequently used evaluation measure for model calibration. We'll then cover some of the drawbacks of this measure and how these surfaced the need for additional notions of calibration, which require their own new evaluation measures. This post is not intended to be an in-depth dissection of all works on calibration, nor does it focus on how to calibrate models. Instead, it is meant to provide a gentle introduction to the different notions and their evaluation measures as well as to re-highlight some issues with a measure that is still widely used to evaluate calibration.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 23 of its 33 references that are in the map. First posted on arXiv 31 Jan 2025.

Builds on

23 of 33 references.

References · 23

  1. 2017On Calibration of Modern Neural Networkscited by 4,860
  2. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  3. 2019Measuring Calibration in Deep Learningcited by 443
  4. 2019Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibrationcited by 359
  5. 2022Prompting GPT-3 To Be Reliablecited by 258
  6. 2019Human uncertainty makes classification more robustcited by 231
  7. 2020Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learningcited by 208
  8. 2021Deep Deterministic Uncertainty: A Simple Baselinecited by 200
  9. 2020Mix-n-Match: Ensemble and Compositional Methods for Uncertainty Calibration in Deep Learningcited by 190
  10. 2019Evaluating model calibration in classificationcited by 165
  11. 2020What Can We Learn from Collective Human Opinions on Natural Language Inference Data?cited by 148
  12. 2020Mitigating Bias in Calibration Error Estimationcited by 100
11 more
  1. 2020Calibration of Neural Networks using Splinescited by 97
  2. 2023DICES Dataset: Diversity in Conversational AI Evaluation for Safetycited by 86
  3. 2019Calibration tests in multi-class classification: A unifying frameworkcited by 79
  4. 2022Stop Measuring Calibration When Humans Disagreecited by 70
  5. 2024SPUQ: Perturbation-Based Uncertainty Quantification for Large Language Modelscited by 54
  6. 2023Human Uncertainty in Concept-Based AI Systemscited by 39
  7. 2022T-Cal: An optimal test for the calibration of predictive modelscited by 24
  8. 2022Ambiguous Images With Human Judgments for Robust Visual Event Classificationcited by 12
  9. 2024Information-theoretic Generalization Analysis for Expected Calibration Errorcited by 9
  10. 2024Ambiguous Annotations: When is a Pedestrian not a Pedestrian?cited by 4
  11. 2024A Confidence Interval for the ℓ2 Expected Calibration Errorcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 13.

20 papers.

Sort:

By area:

  1. 2025Beyond Linear Probes: Dynamic Safety Monitoring for Language Modelscited by 14
  2. 2025Calibration Across Layers: Understanding Calibration Evolution in LLMscited by 9
  3. 2025Bridging Human and LLM Judgments: Understanding and Narrowing the Gapcited by 6
  4. 2026Wired for Overconfidence: A Mechanistic Perspective on Inflated Verbalized Confidence in LLMscited by 4
  5. 2026RewardUQ: A Unified Framework for Uncertainty-Aware Reward Modelscited by 3
  6. 2026I Can't Believe It's Not Robust: Catastrophic Collapse of Safety Classifiers under Embedding Driftcited by 3
  7. 2025Can Confidence Estimates Decide When Chain-of-Thought Is Necessary for LLMs?cited by 2
  8. 2026Confidence Calibration in Large Language Modelscited by 1
  9. 2025Weak Links in LinkedIn: Enhancing Fake Profile Detection in the Age of LLMscited by 0
  10. 2025An Explainable and Fair AI Tool for PCOS Risk Assessment: Calibration, Subgroup Equity, and Interactive Clinical Deploymentcited by 0
  11. 2025How (Mis)calibrated is your Federated CLIP and what to do about it?cited by 0
  12. 2026Improving Detection of Rare Nodes in Hierarchical Multi-Label Learningcited by 0
8 more
  1. 2026MUSE: Multi-Tenant Model Serving With Seamless Model Updatescited by 0
  2. 2026Lean Learning Beyond Clouds: Efficient Discrepancy-Conditioned Optical-SAR Fusion for Semantic Segmentationcited by 0
  3. 2026Detecting Hallucinations in Large Language Models via Internal Attention Divergence Signalscited by 0
  4. 2026Task-Aware Calibration: Provably Optimal Decoding in LLMscited by 0
  5. 2026An Assessment of Human vs. Model Uncertainty in Soft-Label Learning and Calibrationcited by 0
  6. 2026Calibration vs Decision Making: Revisiting the Reliability Paradox in Unlearned Language Modelscited by 0
  7. 2026ST-LoRA: Single Trajectory LoRA Ensemble for Uncertainty Aware Agricultural Segmentationcited by 0
  8. 2026Subspace Inference Enables Efficient Active Reward Learning from Preferencescited by 0