# Honest calibration assessment for binary outcome predictions

arXiv: https://arxiv.org/abs/2203.04065 · HTML: https://mlsanity.com/paper/2203.04065/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Mar 2022.

## Built on by (most cited first, 8 of 8)

- [Isotonic Recalibration under a Low Signal-to-Noise Ratio](https://mlsanity.com/paper/2301.02692/) (2023)
- [Evaluating Probabilistic Classifiers: The Triptych](https://mlsanity.com/paper/2301.10803/) (2023)
- [Calibration Bands for Mean Estimates within the Exponential Dispersion Family](https://mlsanity.com/paper/2503.18896/) (2025)
- [BKP: An R Package for Beta Kernel Process Modeling](https://mlsanity.com/paper/2508.10447/) (2025)
- [Measuring Uncertainty Calibration](https://mlsanity.com/paper/2512.13872/) (2025)
- [Recipes for Calibration Checks in Safety-Critical Applications](https://mlsanity.com/paper/2604.26479/) (2026)
- [Identification and Honest Recovery from Semantic Observation Kernels: Operator Error, Coarsening, and Stability](https://mlsanity.com/paper/2607.23130/) (2026)
- [An analysis of binary isotonic regression: degrees of freedom and implications for calibration](https://mlsanity.com/paper/2607.27301/) (2026)

## Abstract (arXiv)

Probability predictions from binary regressions or machine learning methods ought to be calibrated: If an event is predicted to occur with probability x, it should materialize with approximately that frequency, which means that the so-called calibration curve p(·) should equal the identity, p(x) = x for all x in the unit interval. We propose honest calibration assessment based on novel confidence bands for the calibration curve, which are valid only subject to the natural assumption of isotonicity. Besides testing the classical goodness-of-fit null hypothesis of perfect calibration, our bands facilitate inverted goodness-of-fit tests whose rejection allows for the sought-after conclusion of a sufficiently well specified model. We show that our bands have a finite sample coverage guarantee, are narrower than existing approaches, and adapt to the local smoothness of the calibration curve p and the local variance of the binary observations. In an application to model predictions of an infant having a low birth weight, the bounds give informative insights on model calibration.