mlsanityData to 13 Oct 2026

A Unifying Theory of Distance from Calibration

arXiv 2211.16886 ↗First posted 30 Nov 2022cs.LGcited by 52

Not on the map. Why?

Abstract (arXiv)

We study the fundamental question of how to define and measure the distance from calibration for probabilistic predictors. While the notion of perfect calibration is well-understood, there is no consensus on how to quantify the distance from perfect calibration. Numerous calibration measures have been proposed in the literature, but it is unclear how they compare to each other, and many popular measures such as Expected Calibration Error (ECE) fail to satisfy basic properties like continuity. We present a rigorous framework for analyzing calibration measures, inspired by the literature on property testing. We propose a ground-truth notion of distance from calibration: the ℓ1 distance to the nearest perfectly calibrated predictor. We define a consistent calibration measure as one that is polynomially related to this distance. Applying our framework, we identify three calibration measures that are consistent and can be estimated efficiently: smooth calibration, interval calibration, and Laplace kernel calibration. The former two give quadratic approximations to the ground truth distance, which we show is information-theoretically optimal in a natural model for measuring calibration which we term the prediction-only access model. Our work thus establishes fundamental lower and upper bounds on measuring the distance to calibration, and also provides theoretical justification for preferring certain metrics (like Laplace kernel calibration) in practice.

Key facts. Cited by 52 papers in the map (data through 13 Oct 2026). Builds on 21 of its 64 references that are in the map. First posted on arXiv 30 Nov 2022.

Builds on

21 of 64 references.

References · 21

  1. 2017On Calibration of Modern Neural Networkscited by 4,902
  2. 2016Inherent Trade-Offs in the Fair Determination of Risk Scorescited by 585
  3. 2019Measuring Calibration in Deep Learningcited by 443
  4. 2021Revisiting the Calibration of Modern Neural Networkscited by 388
  5. 2017Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairnesscited by 382
  6. 2019Verified Uncertainty Calibrationcited by 307
  7. 2020Mix-n-Match: Ensemble and Compositional Methods for Uncertainty Calibration in Deep Learningcited by 190
  8. 2020Uncertainty Quantification and Deep Ensemblescited by 138
  9. 2020Mitigating Bias in Calibration Error Estimationcited by 100
  10. 2020Calibration of Neural Networks using Splinescited by 97
  11. 2021Soft Calibration Objectives for Neural Networkscited by 90
  12. 2020Outcome Indistinguishabilitycited by 74
9 more
  1. 2021Omnipredictorscited by 72
  2. 2022Loss Minimization through the Lens of Outcome Indistinguishabilitycited by 62
  3. 2022Low-Degree Multicalibrationcited by 44
  4. 2021Supervised Transfer Learning at Scale for Medical Imagingcited by 38
  5. 2022Metrics of calibration for probabilistic predictionscited by 31
  6. 2022T-Cal: An optimal test for the calibration of predictive modelscited by 24
  7. 2022Smooth Calibration, Leaky Forecasts, Finite Recall, and Nash Dynamicscited by 22
  8. 2022Forecast Hedging and Calibrationcited by 22
  9. 2014Binary Classifier Calibration: Non-parametric approachcited by 13

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 10; 2024: 13; 2025: 16; 2026: 13.

52 papers.

Sort:

By area:

  1. 2024Learning to Route LLMs with Confidence Tokenscited by 67
  2. 2023Smooth ECE: Principled Reliability Diagrams via Kernel Smoothingcited by 61
  3. 2024Conformal Prediction for Natural Language Processing: A Surveycited by 52
  4. 2023U-Calibration: Forecasting for an Unknown Agentcited by 46
  5. 2023When Does Optimizing a Proper Loss Yield Calibration?cited by 36
  6. 2024Towards Data-Centric RLHF: Simple Metrics for Preference Dataset Comparisoncited by 21
  7. 2025From Calibration to Collaboration: LLM Uncertainty Quantification Should Be More Human-Centeredcited by 21
  8. 2023Loss Minimization Yields Multicalibration for Large Neural Networkscited by 19
  9. 2024On Computationally Efficient Multi-Class Calibrationcited by 16
  10. 2024When is Multicalibration Post-Processing Necessary?cited by 16
  11. 2023Kandinsky Conformal Prediction: Efficient Calibration of Image Segmentation Algorithmscited by 14
  12. 2024Optimal Multiclass U-Calibration Error and Beyondcited by 13
38 more
  1. 2025High-Dimensional Calibration from Swap Regretcited by 13
  2. 2024Tractable Agreement Protocolscited by 12
  3. 2023Minimum-Risk Recalibration of Classifierscited by 10
  4. 2024Testing Calibration in Nearly-Linear Timecited by 10
  5. 2024Bridging Multicalibration and Out-of-distribution Generalization Beyond Covariate Shiftcited by 10
  6. 2025Smooth Calibration and Decision Makingcited by 10
  7. 2025How Global Calibration Strengthens Multiaccuracycited by 8
  8. 2023On the Limitations of Temperature Scaling for Distributions with Overlapscited by 7
  9. 2025Three Types of Calibration with Properties and their Semantic and Formal Relationshipscited by 5
  10. 2025In Defense of Defensive Forecastingcited by 5
  11. 2025Improved Bounds for Swap Multicalibration and Swap Omnipredictioncited by 4
  12. 2025Calibration through the Lens of Indistinguishabilitycited by 4
  13. 2024Fully Heteroscedastic Count Regression with Deep Double Poisson Networkscited by 3
  14. 2025Efficient Calibration for Decision Makingcited by 3
  15. 2026The Importance of Being Smoothly Calibratedcited by 3
  16. 2023On the Within-Group Fairness of Screening Classifierscited by 2
  17. 2023Set Learning for Accurate and Calibrated Modelscited by 2
  18. 2026Testable and Actionable Calibration for Full Swap Regretcited by 2
  19. 2023An Operational Perspective to Fairness Interventions: Where and How to Intervenecited by 1
  20. 2024An Elementary Predictor Obtaining 2√T+1 Distance to Calibrationcited by 1
  21. 2024Combining Priors with Experience: Confidence Calibration Based on Binomial Process Modelingcited by 1
  22. 2025Probability Bounding: Post-Hoc Calibration via Box-Constrained Softmaxcited by 1
  23. 2025Evaluating the Quality of the Quantified Uncertainty for (Re)Calibration of Data-Driven Regression Modelscited by 1
  24. 2026Computational and Statistical Hardness of Calibration Distancecited by 1
  25. 2026The Verification Tax: Fundamental Limits of AI Auditing in the Rare-Error Regimecited by 1
  26. 2026Budgeted Act-or-Defer Multi-Agent LLM Deliberation with Local Reliability Boundscited by 1
  27. 2024Assessing the Probabilistic Fit of Neural Regressors via Conditional Congruencecited by 0
  28. 2025Uniform convergence of the smooth calibration error and its relationship with functional gradientcited by 0
  29. 2025Auditability and the Landscape of Distance to Multicalibrationcited by 0
  30. 2025L2-Regularized Empirical Risk Minimization Guarantees Small Smooth Calibration Errorcited by 0
  31. 2025Scalable Utility-Aware Multiclass Calibrationcited by 0
  32. 2025SimClinician: A Multimodal Simulation Testbed for Reliable Psychologist AI Collaboration in Mental Health Diagnosiscited by 0
  33. 2026The Minimax Rate of Perturbed Second-Order Calibrationcited by 0
  34. 2026Smoothed Elicitation Complexity for Approximate Gamma-calibration of Discrete Classification Taskscited by 0
  35. 2026Enhancing Deep Neural Network Reliability with Refinement and Calibrationcited by 0
  36. 2026Human-AI Teaming Through the Lens of Calibrationcited by 0
  37. 2026An analysis of binary isotonic regression: degrees of freedom and implications for calibrationcited by 0
  38. 2026Truthful Calibration Measures for Sequential Predictioncited by 0

50 most cited of 52 All 52 (JSON)