# Evaluating probabilistic classifiers: Reliability diagrams and score decompositions revisited

arXiv: https://arxiv.org/abs/2008.03033 · HTML: https://mlsanity.com/paper/2008.03033/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Aug 2020.

## Built on by (most cited first, 26 of 26)

- [Smooth ECE: Principled Reliability Diagrams via Kernel Smoothing](https://mlsanity.com/paper/2309.12236/) (2023)
- [Rethinking Early Stopping: Refine, Then Calibrate](https://mlsanity.com/paper/2501.19195/) (2025)
- [Calibrated Decisions at Scale: Converting Police Crash Narratives into Probabilistic Crash Variables with a System One Model (Jev)](https://mlsanity.com/paper/2609.24052/) (2026)
- [Model Comparison and Calibration Assessment: User Guide for Consistent Scoring Functions in Machine Learning and Actuarial Practice](https://mlsanity.com/paper/2202.12780/) (2022)
- [On (assessing) the fairness of risk score models](https://mlsanity.com/paper/2302.08851/) (2023)
- [Consistent and Asymptotically Unbiased Estimation of Proper Calibration Errors](https://mlsanity.com/paper/2312.08589/) (2023)
- [Revisiting Confidence Estimation: Towards Reliable Failure Prediction](https://mlsanity.com/paper/2403.02886/) (2024)
- [Evaluating Probabilistic Classifiers: The Triptych](https://mlsanity.com/paper/2301.10803/) (2023)
- [Calibrate: Interactive Analysis of Probabilistic Model Output](https://mlsanity.com/paper/2207.13770/) (2022)
- [How Flawed Is ECE? An Analysis via Logit Smoothing](https://mlsanity.com/paper/2402.10046/) (2024)
- [TCE: A Test-Based Approach to Measuring Calibration Error](https://mlsanity.com/paper/2306.14343/) (2023)
- [Improving Semantic Uncertainty Quantification in Language Model Question-Answering via Token-Level Temperature Scaling](https://mlsanity.com/paper/2604.07172/) (2026)
- [Hierarchies of Calibration: Classification meets Regression](https://mlsanity.com/paper/2606.03245/) (2026)
- [Verifiable Rewards for Calibrated Probabilistic Forecasting](https://mlsanity.com/paper/2607.00164/) (2026)
- [Boldness-Recalibration for Binary Event Predictions](https://mlsanity.com/paper/2305.03780/) (2023)
- [When Pattern-by-Pattern Works: Theoretical and Empirical Insights for Logistic Models with Missing Values](https://mlsanity.com/paper/2507.13024/) (2025)
- [Proper Calibeating](https://mlsanity.com/paper/2605.26703/) (2026)
- [Fear Is Not Always Fear: Variation in Emotion Label Equivalence Across Datasets](https://mlsanity.com/paper/2610.12841/) (2026)
- [Towards a data-driven debt collection strategy based on an advanced machine learning framework](https://mlsanity.com/paper/2311.06292/) (2023)
- [The distribution of calibrated likelihood functions on the probability-likelihood Aitchison simplex](https://mlsanity.com/paper/2509.03365/) (2025)
- [SeFEF: A Seizure Forecasting Evaluation Framework](https://mlsanity.com/paper/2510.11275/) (2025)
- [V-ECE: Estimating General Expected Calibration Errors](https://mlsanity.com/paper/2602.24230/) (2026)
- [Decomposing Probabilistic Scores: Reliability, Information Loss and Uncertainty](https://mlsanity.com/paper/2603.15232/) (2026)
- [Bayesian Complete-Pooling in Cross-Subject Classification for Motor Imagery Electroencephalogram](https://mlsanity.com/paper/2607.22980/) (2026)
- [ST-LoRA: Single Trajectory LoRA Ensemble for Uncertainty Aware Agricultural Segmentation](https://mlsanity.com/paper/2608.01530/) (2026)
- [Suitable Measures for the Potential Operational Utility of AI NWP Rainfall Forecasts Over Africa](https://mlsanity.com/paper/2609.31775/) (2026)

## Abstract (arXiv)

A probability forecast or probabilistic classifier is reliable or calibrated if the predicted probabilities are matched by ex post observed frequencies, as examined visually in reliability diagrams. The classical binning and counting approach to plotting reliability diagrams has been hampered by a lack of stability under unavoidable, ad hoc implementation decisions. Here we introduce the CORP approach, which generates provably statistically Consistent, Optimally binned, and Reproducible reliability diagrams in an automated way. CORP is based on non-parametric isotonic regression and implemented via the Pool-adjacent-violators (PAV) algorithm - essentially, the CORP reliability diagram shows the graph of the PAV- (re)calibrated forecast probabilities. The CORP approach allows for uncertainty quantification via either resampling techniques or asymptotic theory, furnishes a new numerical measure of miscalibration, and provides a CORP based Brier score decomposition that generalizes to any proper scoring rule. We anticipate that judicious uses of the PAV algorithm yield improved tools for diagnostics and inference for a very wide range of statistical and machine learning methods.