# Decomposing Probabilistic Scores: Reliability, Information Loss and Uncertainty

arXiv: https://arxiv.org/abs/2603.15232 · HTML: https://mlsanity.com/paper/2603.15232/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 25 references that are in the map.
- First posted on arXiv 16 Mar 2026.

## Builds on (most cited first, 7 of 7)

- [Mix-n-Match: Ensemble and Compositional Methods for Uncertainty Calibration in Deep Learning](https://mlsanity.com/paper/2003.07329/) (2020)
- [Calibration of Neural Networks using Splines](https://mlsanity.com/paper/2006.12800/) (2020)
- [Beyond calibration: estimating the grouping loss of modern neural networks](https://mlsanity.com/paper/2210.16315/) (2022)
- [Evaluating probabilistic classifiers: Reliability diagrams and score decompositions revisited](https://mlsanity.com/paper/2008.03033/) (2020)
- [Rethinking Early Stopping: Refine, Then Calibrate](https://mlsanity.com/paper/2501.19195/) (2025)
- [Calibrating sufficiently](https://mlsanity.com/paper/2105.07283/) (2021)
- [Evaluating Probabilistic Classifiers: The Triptych](https://mlsanity.com/paper/2301.10803/) (2023)

## Abstract (arXiv)

Calibration is a conditional property that depends on the information retained by a predictor. We develop decomposition identities for arbitrary proper losses that make this dependence explicit. At any information level mathcal A, the expected loss of an mathcal A-measurable predictor splits into a proper-regret (reliability) term and a conditional entropy (residual uncertainty) term. For nested levels mathcal Asubseteqmathcal B, a chain decomposition quantifies the information gain from mathcal A to mathcal B. Applied to classification with features boldsymbolX and score S=s(boldsymbolX), this yields a three-term identity: miscalibration, a em grouping term measuring information loss from boldsymbolX to S, and irreducible uncertainty at the feature level. We leverage the framework to analyze post-hoc recalibration, aggregation of calibrated models, and stagewise/boosting constructions, with explicit forms for Brier and log-loss.