# Metrics of calibration for probabilistic predictions

arXiv: https://arxiv.org/abs/2205.09680 · HTML: https://mlsanity.com/paper/2205.09680/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 19 references that are in the map.
- First posted on arXiv 19 May 2022.

## Builds on (most cited first, 12 of 12)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Measuring Calibration in Deep Learning](https://mlsanity.com/paper/1904.01685/) (2019)
- [Verified Uncertainty Calibration](https://mlsanity.com/paper/1909.10155/) (2019)
- [Evaluating model calibration in classification](https://mlsanity.com/paper/1902.06977/) (2019)
- [Mitigating Bias in Calibration Error Estimation](https://mlsanity.com/paper/2012.08668/) (2020)
- [Calibration of Neural Networks using Splines](https://mlsanity.com/paper/2006.12800/) (2020)
- [T-Cal: An optimal test for the calibration of predictive models](https://mlsanity.com/paper/2203.01850/) (2022)
- [Calibration of P-values for calibration and for deviation of a subpopulation from the full population](https://mlsanity.com/paper/2202.00100/) (2022)
- [Cumulative deviation of a subpopulation from the full population](https://mlsanity.com/paper/2008.01779/) (2020)
- [A graphical method of cumulative differences between two subpopulations](https://mlsanity.com/paper/2108.02666/) (2021)

## Built on by (most cited first, 30 of 32)

- [Smooth ECE: Principled Reliability Diagrams via Kernel Smoothing](https://mlsanity.com/paper/2309.12236/) (2023)
- [A Unifying Theory of Distance from Calibration](https://mlsanity.com/paper/2211.16886/) (2022)
- [Can a calibration metric be both testable and actionable?](https://mlsanity.com/paper/2502.19851/) (2025)
- [Calibrating LLM Judges: Linear Probes for Fast and Reliable Uncertainty Estimation](https://mlsanity.com/paper/2512.22245/) (2025)
- [Evaluating Probabilistic Classifiers: The Triptych](https://mlsanity.com/paper/2301.10803/) (2023)
- [Reassessing How to Compare and Improve the Calibration of Machine Learning Models](https://mlsanity.com/paper/2406.04068/) (2024)
- [A Survey of Calibration Process for Black-Box LLMs](https://mlsanity.com/paper/2412.12767/) (2024)
- [A Kernel-Based Conditional Two-Sample Test Using Nearest Neighbors (with Applications to Calibration, Regression Curves, and Simulation-Based Inference)](https://mlsanity.com/paper/2407.16550/) (2024)
- [How Flawed Is ECE? An Analysis via Logit Smoothing](https://mlsanity.com/paper/2402.10046/) (2024)
- [TCE: A Test-Based Approach to Measuring Calibration Error](https://mlsanity.com/paper/2306.14343/) (2023)
- [MCGrad: Multicalibration at Web Scale](https://mlsanity.com/paper/2509.19884/) (2025)
- [Measuring multi-calibration](https://mlsanity.com/paper/2506.11251/) (2025)
- [Labels in Extremes: How Well Calibrated are Extreme Multi-label Classifiers?](https://mlsanity.com/paper/2411.04276/) (2024)
- [Multiclass Local Calibration with the Jensen-Shannon Distance](https://mlsanity.com/paper/2510.26566/) (2025)
- [Concepts Worth Having: Refining VLM-Guided Concept Bottleneck Models with Minimal Annotations](https://mlsanity.com/paper/2605.16405/) (2026)
- [Testable and Actionable Calibration for Full Swap Regret](https://mlsanity.com/paper/2605.17749/) (2026)
- [CalArena: A Large-Scale Post-Hoc Calibration Benchmark](https://mlsanity.com/paper/2605.30188/) (2026)
- [Decision Making Needs Uncertainty Quantification [Lecture Notes]](https://mlsanity.com/paper/2607.14407/) (2026)
- [An Operational Perspective to Fairness Interventions: Where and How to Intervene](https://mlsanity.com/paper/2302.01574/) (2023)
- [Measuring Uncertainty Calibration](https://mlsanity.com/paper/2512.13872/) (2025)
- [Beyond ECE: Calibrated Size Ratio, Risk Assessment, and Confidence-Weighted Metrics](https://mlsanity.com/paper/2605.01796/) (2026)
- [Divide et Calibra: Multiclass Local Calibration via Vector Quantization](https://mlsanity.com/paper/2605.21060/) (2026)
- [Towards a data-driven debt collection strategy based on an advanced machine learning framework](https://mlsanity.com/paper/2311.06292/) (2023)
- [Uniform convergence of the smooth calibration error and its relationship with functional gradient](https://mlsanity.com/paper/2505.19396/) (2025)
- [Calibration improves detection of mislabeled examples](https://mlsanity.com/paper/2511.02738/) (2025)
- [SimClinician: A Multimodal Simulation Testbed for Reliable Psychologist AI Collaboration in Mental Health Diagnosis](https://mlsanity.com/paper/2512.08953/) (2025)
- [VOLTA: The Surprising Ineffectiveness of Auxiliary Losses for Calibrated Deep Learning](https://mlsanity.com/paper/2604.08639/) (2026)
- [Multicalibration for Unbiased Model-Based Prevalence Estimation](https://mlsanity.com/paper/2604.21549/) (2026)
- [Uncertainty quantification via conformal prediction in data assimilation](https://mlsanity.com/paper/2606.27001/) (2026)
- [Bandwidth Selection in Kernel Density Estimation for Model Calibration](https://mlsanity.com/paper/2606.29925/) (2026)

## Abstract (arXiv)

Predictions are often probabilities; e.g., a prediction could be for precipitation tomorrow, but with only a 30% chance. Given such probabilistic predictions together with the actual outcomes, "reliability diagrams" help detect and diagnose statistically significant discrepancies -- so-called "miscalibration" -- between the predictions and the outcomes. The canonical reliability diagrams histogram the observed and expected values of the predictions; replacing the hard histogram binning with soft kernel density estimation is another common practice. But, which widths of bins or kernels are best? Plots of the cumulative differences between the observed and expected values largely avoid this question, by displaying miscalibration directly as the slopes of secant lines for the graphs. Slope is easy to perceive with quantitative precision, even when the constant offsets of the secant lines are irrelevant; there is no need to bin or perform kernel density estimation. The existing standard metrics of miscalibration each summarize a reliability diagram as a single scalar statistic. The cumulative plots naturally lead to scalar metrics for the deviation of the graph of cumulative differences away from zero; good calibration corresponds to a horizontal, flat graph which deviates little from zero. The cumulative approach is currently unconventional, yet offers many favorable statistical properties, guaranteed via mathematical theory backed by rigorous proofs and illustrative numerical examples. In particular, metrics based on binning or kernel density estimation unavoidably must trade-off statistical confidence for the ability to resolve variations as a function of the predicted probability or vice versa. Widening the bins or kernels averages away random noise while giving up some resolving power. Narrowing the bins or kernels enhances resolving power while not averaging away as much noise.