# Measuring Uncertainty Calibration

arXiv: https://arxiv.org/abs/2512.13872 · HTML: https://mlsanity.com/paper/2512.13872/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 36 references that are in the map.
- First posted on arXiv 15 Dec 2025.

## Builds on (most cited first, 17 of 17)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Measuring Calibration in Deep Learning](https://mlsanity.com/paper/1904.01685/) (2019)
- [Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classification](https://mlsanity.com/paper/1903.04561/) (2019)
- [Verified Uncertainty Calibration](https://mlsanity.com/paper/1909.10155/) (2019)
- [Mix-n-Match: Ensemble and Compositional Methods for Uncertainty Calibration in Deep Learning](https://mlsanity.com/paper/2003.07329/) (2020)
- [Evaluating model calibration in classification](https://mlsanity.com/paper/1902.06977/) (2019)
- [From Amateurs to Connoisseurs: Modeling the Evolution of User Expertise through Online Reviews](https://mlsanity.com/paper/1303.4402/) (2013)
- [Distribution-free binary classification: prediction sets, confidence intervals and calibration](https://mlsanity.com/paper/2006.10564/) (2020)
- [Metrics of calibration for probabilistic predictions](https://mlsanity.com/paper/2205.09680/) (2022)
- [T-Cal: An optimal test for the calibration of predictive models](https://mlsanity.com/paper/2203.01850/) (2022)
- [Modular proximal optimization for multidimensional total-variation regularization](https://mlsanity.com/paper/1411.0589/) (2014)
- [Information-theoretic Generalization Analysis for Expected Calibration Error](https://mlsanity.com/paper/2405.15709/) (2024)
- [Honest calibration assessment for binary outcome predictions](https://mlsanity.com/paper/2203.04065/) (2022)
- [Probability Calibration Trees](https://mlsanity.com/paper/1808.00111/) (2018)
- [Calibration of P-values for calibration and for deviation of a subpopulation from the full population](https://mlsanity.com/paper/2202.00100/) (2022)
- [On the Usefulness of the Fit-on-the-Test View on Evaluating Calibration of Classifiers](https://mlsanity.com/paper/2203.08958/) (2022)

## Built on by (most cited first, 1 of 1)

- [The Minimax Rate of Perturbed Second-Order Calibration](https://mlsanity.com/paper/2605.07808/) (2026)

## Abstract (arXiv)

We make two contributions to the problem of estimating the L1 calibration error of a binary classifier from a finite dataset. First, we provide an upper bound for any classifier where the calibration function has bounded variation. Second, we provide a method of modifying any classifier so that its calibration error can be upper bounded efficiently without significantly impacting classifier performance and without any restrictive assumptions. All our results are non-asymptotic and distribution-free. We conclude by providing advice on how to measure calibration error in practice. Our methods yield practical procedures that can be run on real-world datasets with modest overhead.