mlsanityData to 7 Oct 2026

Measuring Uncertainty Calibration

arXiv 2512.13872 ↗First posted 15 Dec 2025cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

We make two contributions to the problem of estimating the L1 calibration error of a binary classifier from a finite dataset. First, we provide an upper bound for any classifier where the calibration function has bounded variation. Second, we provide a method of modifying any classifier so that its calibration error can be upper bounded efficiently without significantly impacting classifier performance and without any restrictive assumptions. All our results are non-asymptotic and distribution-free. We conclude by providing advice on how to measure calibration error in practice. Our methods yield practical procedures that can be run on real-world datasets with modest overhead.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 17 of its 36 references that are in the map. First posted on arXiv 15 Dec 2025.

Builds on

17 of 36 references.

References · 17

  1. 2017On Calibration of Modern Neural Networkscited by 4,860
  2. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  3. 2019Measuring Calibration in Deep Learningcited by 443
  4. 2019Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classificationcited by 396
  5. 2019Verified Uncertainty Calibrationcited by 307
  6. 2020Mix-n-Match: Ensemble and Compositional Methods for Uncertainty Calibration in Deep Learningcited by 190
  7. 2019Evaluating model calibration in classificationcited by 165
  8. 2013From Amateurs to Connoisseurs: Modeling the Evolution of User Expertise through Online Reviewscited by 107
  9. 2020Distribution-free binary classification: prediction sets, confidence intervals and calibrationcited by 79
  10. 2022Metrics of calibration for probabilistic predictionscited by 31
  11. 2022T-Cal: An optimal test for the calibration of predictive modelscited by 24
  12. 2014Modular proximal optimization for multidimensional total-variation regularizationcited by 10
5 more
  1. 2024Information-theoretic Generalization Analysis for Expected Calibration Errorcited by 9
  2. 2022Honest calibration assessment for binary outcome predictionscited by 8
  3. 2018Probability Calibration Treescited by 6
  4. 2022Calibration of P-values for calibration and for deviation of a subpopulation from the full populationcited by 4
  5. 2022On the Usefulness of the Fit-on-the-Test View on Evaluating Calibration of Classifierscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026The Minimax Rate of Perturbed Second-Order Calibrationcited by 0