Decomposing Probabilistic Scores: Reliability, Information Loss and Uncertainty

Calibration is a conditional property that depends on the information retained by a predictor. We develop decomposition identities for arbitrary proper losses that make this dependence explicit. At any information level mathcal A, the expected loss of an mathcal A-measurable predictor splits into a proper-regret (reliability) term and a conditional entropy (residual uncertainty) term. For nested levels mathcal Asubseteqmathcal B, a chain decomposition quantifies the information gain from mathcal A to mathcal B. Applied to classification with features boldsymbolX and score S=s(boldsymbolX), this yields a three-term identity: miscalibration, a em grouping term measuring information loss from boldsymbolX to S, and irreducible uncertainty at the feature level. We leverage the framework to analyze post-hoc recalibration, aggregation of calibrated models, and stagewise/boosting constructions, with explicit forms for Brier and log-loss.