# Estimating Expected Calibration Errors

arXiv: https://arxiv.org/abs/2109.03480 · HTML: https://mlsanity.com/paper/2109.03480/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 17 references that are in the map.
- First posted on arXiv 8 Sep 2021.

## Builds on (most cited first, 5 of 5)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Measuring Calibration in Deep Learning](https://mlsanity.com/paper/1904.01685/) (2019)
- [Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration](https://mlsanity.com/paper/1910.12656/) (2019)
- [Distribution Calibration for Regression](https://mlsanity.com/paper/1905.06023/) (2019)
- [Calibrated Prediction Intervals for Neural Network Regressors](https://mlsanity.com/paper/1803.09546/) (2018)

## Built on by (most cited first, 24 of 24)

- [Robustness in Large Language Models: A Survey of Mitigation Strategies and Evaluation Metrics](https://mlsanity.com/paper/2505.18658/) (2025)
- [Confidence-based Estimators for Predictive Performance in Model Monitoring](https://mlsanity.com/paper/2407.08649/) (2024)
- [Active Learning and Novel Model Calibration Measurements for Automated Visual Inspection in Manufacturing](https://mlsanity.com/paper/2209.05486/) (2022)
- [Calibrate to Interpret](https://mlsanity.com/paper/2207.03324/) (2022)
- [Machine learning for sports betting: should model selection be based on accuracy or calibration?](https://mlsanity.com/paper/2303.06021/) (2023)
- [A Measure of the System Dependence of Automated Metrics](https://mlsanity.com/paper/2412.03152/) (2024)
- [Performance Estimation in Binary Classification Using Calibrated Confidence](https://mlsanity.com/paper/2505.05295/) (2025)
- [On the Risk of Misleading Reports: Diagnosing Textual Biases in Multimodal Clinical AI](https://mlsanity.com/paper/2508.00171/) (2025)
- [Mechanistic Circuit Identification for Controllable Data Generation](https://mlsanity.com/paper/2608.24065/) (2026)
- [Prosocial Behavior Detection in Player Game Chat: From Aligning Human-AI Definitions to Efficient Annotation at Scale](https://mlsanity.com/paper/2508.05938/) (2025)
- [Trajectory-Aware Reliability Modeling of Democratic Systems](https://mlsanity.com/paper/2604.20127/) (2026)
- [Advancing sleep detection by modelling weak label sets: A novel weakly supervised learning approach](https://mlsanity.com/paper/2402.17601/) (2024)
- [The Bayesian Confidence (BACON) Estimator for Deep Neural Networks](https://mlsanity.com/paper/2410.12604/) (2024)
- [Ensemble BERT for Medication Event Classification on Electronic Health Records (EHRs)](https://mlsanity.com/paper/2506.23315/) (2025)
- [Causal Understanding by LLMs: The Role of Uncertainty](https://mlsanity.com/paper/2509.20088/) (2025)
- [Multiclass Calibration Assessment and Recalibration of Probability Predictions via the Linear Log Odds Calibration Function](https://mlsanity.com/paper/2602.18573/) (2026)
- [CHiL(L)Grader: Calibrated Human-in-the-Loop Short-Answer Grading](https://mlsanity.com/paper/2603.11957/) (2026)
- [Clinically Aware Synthetic Image Generation for Concept Coverage in Chest X-ray Models](https://mlsanity.com/paper/2603.15525/) (2026)
- [Enhancing the Reliability of Medical AI through Expert-guided Uncertainty Modeling](https://mlsanity.com/paper/2604.01898/) (2026)
- [EviRCOD: Evidence-Guided Probabilistic Decoding for Referring Camouflaged Object Detection](https://mlsanity.com/paper/2604.10894/) (2026)
- [Rethinking Vacuity for OOD Detection in Evidential Deep Learning](https://mlsanity.com/paper/2605.06382/) (2026)
- [The Critical Role of Model Selection in Causal Inference: A Comparative Analysis of Classification Models within the InferBERT Framework for Pharmacovigilance](https://mlsanity.com/paper/2606.17113/) (2026)
- [ResLearn-XR: Residual Learning for Network Traffic and Quality-of-Experience-Aware Modeling in Extended Reality](https://mlsanity.com/paper/2609.04493/) (2026)
- [How to Estimate Whether You Have Found Several Needles in a Haystack: Measuring Calibration in Multi-Label Text Classification](https://mlsanity.com/paper/2609.26468/) (2026)

## Abstract (arXiv)

Uncertainty in probabilistic classifiers predictions is a key concern when models are used to support human decision making, in broader probabilistic pipelines or when sensitive automatic decisions have to be taken. Studies have shown that most models are not intrinsically well calibrated, meaning that their decision scores are not consistent with posterior probabilities. Hence being able to calibrate these models, or enforce calibration while learning them, has regained interest in recent literature. In this context, properly assessing calibration is paramount to quantify new contributions tackling calibration. However, there is room for improvement for commonly used metrics and evaluation of calibration could benefit from deeper analyses. Thus this paper focuses on the empirical evaluation of calibration metrics in the context of classification. More specifically it evaluates different estimators of the Expected Calibration Error (ECE), amongst which legacy estimators and some novel ones, proposed in this paper. We build an empirical procedure to quantify the quality of these ECE estimators, and use it to decide which estimator should be used in practice for different settings.