# Hierarchies of Calibration: Classification meets Regression

arXiv: https://arxiv.org/abs/2606.03245 · HTML: https://mlsanity.com/paper/2606.03245/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 76 references that are in the map.
- First posted on arXiv 2 Jun 2026.

## Builds on (most cited first, 30 of 32)

- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Measuring Calibration in Deep Learning](https://mlsanity.com/paper/1904.01685/) (2019)
- [Accurate Uncertainties for Deep Learning Using Calibrated Regression](https://mlsanity.com/paper/1807.00263/) (2018)
- [Revisiting the Calibration of Modern Neural Networks](https://mlsanity.com/paper/2106.07998/) (2021)
- [Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration](https://mlsanity.com/paper/1910.12656/) (2019)
- [Verified Uncertainty Calibration](https://mlsanity.com/paper/1909.10155/) (2019)
- [Evaluating model calibration in classification](https://mlsanity.com/paper/1902.06977/) (2019)
- [Distribution Calibration for Regression](https://mlsanity.com/paper/1905.06023/) (2019)
- [Calibrating Predictions to Decisions: A Novel Approach to Multi-Class Calibration](https://mlsanity.com/paper/2107.05719/) (2021)
- [Classifier Calibration: A survey on how to assess and improve predicted class probabilities](https://mlsanity.com/paper/2112.10327/) (2021)
- [Better Uncertainty Calibration via Proper Scores for Classification and Beyond](https://mlsanity.com/paper/2203.07835/) (2022)
- [Moment Multicalibration for Uncertainty Estimation](https://mlsanity.com/paper/2008.08037/) (2020)
- [Top-label calibration and multiclass-to-binary reductions](https://mlsanity.com/paper/2107.08353/) (2021)
- [Regression Diagnostics meets Forecast Evaluation: Conditional Calibration, Reliability Diagrams, and Coefficient of Determination](https://mlsanity.com/paper/2108.03210/) (2021)
- [Isotonic Distributional Regression](https://mlsanity.com/paper/1909.03725/) (2019)
- [A Large-Scale Study of Probabilistic Calibration in Neural Network Regression](https://mlsanity.com/paper/2306.02738/) (2023)
- [Evaluating probabilistic classifiers: Reliability diagrams and score decompositions revisited](https://mlsanity.com/paper/2008.03033/) (2020)
- [Calibration by Distribution Matching: Trainable Kernel Calibration Metrics](https://mlsanity.com/paper/2310.20211/) (2023)
- [Generic Conditions for Forecast Dominance](https://mlsanity.com/paper/1805.09902/) (2018)
- [Towards Instance-Wise Calibration: Local Amortized Diagnostics and Reshaping of Conditional Densities (LADaR)](https://mlsanity.com/paper/2205.14568/) (2022)
- [In-sample calibration yields conformal calibration guarantees](https://mlsanity.com/paper/2503.03841/) (2025)
- [Decompositions of the mean continuous ranked probability score](https://mlsanity.com/paper/2311.14122/) (2023)
- [Forecast Evaluation of Quantiles, Prediction Intervals, and other Set-Valued Functionals](https://mlsanity.com/paper/1910.07912/) (2019)
- [Tail calibration of probabilistic forecasts](https://mlsanity.com/paper/2407.03167/) (2024)
- [Assessing the calibration of high-dimensional ensemble forecasts using rank histograms](https://mlsanity.com/paper/1310.0236/) (2013)
- [Score-based calibration testing for multivariate forecast distributions](https://mlsanity.com/paper/2211.16362/) (2022)
- [Assessing the calibration of multivariate probabilistic forecasts](https://mlsanity.com/paper/2307.05846/) (2023)
- [Three Types of Calibration with Properties and their Semantic and Formal Relationships](https://mlsanity.com/paper/2504.18395/) (2025)
- [Calibration through the Lens of Indistinguishability](https://mlsanity.com/paper/2509.02279/) (2025)

## Built on by (most cited first, 2 of 2)

- [Calibration Assessment for Nested Uncertainty Sets](https://mlsanity.com/paper/2610.04790/) (2026)
- [Towards Calibrated Probabilistic Forecasts for Events of Interest via Outcome-Conditional Recalibration](https://mlsanity.com/paper/2610.10076/) (2026)

## Abstract (arXiv)

Concepts of calibration formalize the compatibility between probabilistic predictions and the respective outcomes. In a nutshell, the outcomes ought to be indistinguishable from random draws from the predictive distributions. In this paper, we review, extend, and bridge notions of calibration that have been proposed for classification and regression tasks. Particular emphasis is given to hierarchical relations between the various notions, as they apply to general real-valued data, continuous outcomes, count data, nominal classes, and binary outcomes. To highlight a number of contributions, we introduce the notion of modal calibration for nominal outcomes, we distinguish full, partial, and average calibration in this setting, and we show that double probability integral transform (PIT) calibration is logically independent of previously proposed concepts of calibration for discrete outcomes. Furthermore, we generalize extant results on concepts of calibration that are expressed in terms of properties or functionals of the predictive distributions, such as means, quantiles, or event probabilities. Throughout the paper, we illustrate the concepts and their hierarchical relations in worked examples, and we provide algorithmic tools that support the construction of instructive examples and counterexamples.