mlsanityData to 14 Oct 2026

Can a calibration metric be both testable and actionable?

arXiv 2502.19851 ↗First posted 27 Feb 2025stat.ME stat.MLcited by 12

Not on the map. Why?

Abstract (arXiv)

Forecast probabilities often serve as critical inputs for binary decision making. In such settings, calibrationunicodex2014ensuring forecasted probabilities match empirical frequenciesunicodex2014is essential. Although the common notion of Expected Calibration Error (ECE) provides actionable insights for decision making, it is not testable: it cannot be empirically estimated in many practical cases. Conversely, the recently proposed Distance from Calibration (dCE) is testable, but it is not actionable since it lacks decision-theoretic guarantees needed for high-stakes applications. To resolve this question, we consider Cutoff Calibration Error, a calibration measure that bridges this gap by assessing calibration over intervals of forecasted probabilities. We show that Cutoff Calibration Error is both testable and actionable, and we examine its implications for popular post-hoc calibration methods, such as isotonic regression and Platt scaling.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 14 of its 32 references that are in the map. First posted on arXiv 27 Feb 2025.

Builds on

14 of 32 references.

References · 14

  1. 2017On Calibration of Modern Neural Networkscited by 4,960
  2. 2019Measuring Calibration in Deep Learningcited by 446
  3. 2019Verified Uncertainty Calibrationcited by 309
  4. 2018Multiaccuracy: Black-Box Post-Processing for Fairness in Classificationcited by 222
  5. 2020Mitigating Bias in Calibration Error Estimationcited by 101
  6. 2020Distribution-free binary classification: prediction sets, confidence intervals and calibrationcited by 79
  7. 2023Smooth ECE: Principled Reliability Diagrams via Kernel Smoothingcited by 63
  8. 2023U-Calibration: Forecasting for an Unknown Agentcited by 46
  9. 2023High-Dimensional Prediction for Sequential Decision Makingcited by 41
  10. 2025Near-Optimal Algorithms for Omnipredictioncited by 33
  11. 2022Metrics of calibration for probabilistic predictionscited by 32
  12. 2020Optimization of Scoring Rulescited by 28
2 more
  1. 2024Forecasting for Swap Regret for All Downstream Agentscited by 25
  2. 2024Calibration Error for Decision Makingcited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 7.

12 papers.

  1. 2025Calibration through the Lens of Indistinguishabilitycited by 4
  2. 2025Efficient Calibration for Decision Makingcited by 3
  3. 2025Sample-efficient Multiclass Calibration under ℓp Errorcited by 1
  4. 2025Scalable Utility-Aware Multiclass Calibrationcited by 1
  5. 2026Computational and Statistical Hardness of Calibration Distancecited by 1
  6. 2025Clustered Calibration: Representation-Aware Probability Calibration via Learned Subpopulationscited by 0
  7. 2026Smoothed Elicitation Complexity for Approximate Gamma-calibration of Discrete Classification Taskscited by 0
  8. 2026Human-AI Teaming Through the Lens of Calibrationcited by 0
  9. 2026Calibration without labels in multiple testingcited by 0
  10. 2026An analysis of binary isotonic regression: degrees of freedom and implications for calibrationcited by 0
  11. 2026Sharp Integrality Gaps in Calibration Distancecited by 0
  12. 2026Noisy-Label Calibration with Multiple Unreliable Labelscited by 0