# Regression Diagnostics meets Forecast Evaluation: Conditional Calibration, Reliability Diagrams, and Coefficient of Determination

arXiv: https://arxiv.org/abs/2108.03210 · HTML: https://mlsanity.com/paper/2108.03210/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Aug 2021.

## Built on by (most cited first, 30 of 31)

- [A Large-Scale Study of Probabilistic Calibration in Neural Network Regression](https://mlsanity.com/paper/2306.02738/) (2023)
- [Calibrated Decisions at Scale: Converting Police Crash Narratives into Probabilistic Crash Variables with a System One Model (Jev)](https://mlsanity.com/paper/2609.24052/) (2026)
- [Model Comparison and Calibration Assessment: User Guide for Consistent Scoring Functions in Machine Learning and Actuarial Practice](https://mlsanity.com/paper/2202.12780/) (2022)
- [Isotonic Recalibration under a Low Signal-to-Noise Ratio](https://mlsanity.com/paper/2301.02692/) (2023)
- [Evaluating Probabilistic Classifiers: The Triptych](https://mlsanity.com/paper/2301.10803/) (2023)
- [Reassessing How to Compare and Improve the Calibration of Machine Learning Models](https://mlsanity.com/paper/2406.04068/) (2024)
- [Neural networks for insurance pricing with frequency and severity data: a benchmark study from data preprocessing to technical tariff](https://mlsanity.com/paper/2310.12671/) (2023)
- [Generalized Venn and Venn-Abers Calibration with Applications in Conformal Prediction](https://mlsanity.com/paper/2502.05676/) (2025)
- [From Classification Accuracy to Proper Scoring Rules: Elicitability of Probabilistic Top List Predictions](https://mlsanity.com/paper/2301.11797/) (2023)
- [Three Types of Calibration with Properties and their Semantic and Formal Relationships](https://mlsanity.com/paper/2504.18395/) (2025)
- [Enforcing tail calibration when training probabilistic forecast models](https://mlsanity.com/paper/2506.13687/) (2025)
- [Ensemble learning for uncertainty estimation with application to the correction of satellite precipitation products](https://mlsanity.com/paper/2403.10567/) (2024)
- [Multivariate Latent Recalibration for Conditional Normalizing Flows](https://mlsanity.com/paper/2505.16636/) (2025)
- [Distributional Regression with Tabular Foundation Models: Evaluating Probabilistic Predictions via Proper Scoring Rules](https://mlsanity.com/paper/2603.08206/) (2026)
- [Improving probabilistic forecasts of extreme wind speeds by training statistical post-processing models with weighted scoring rules](https://mlsanity.com/paper/2407.15900/) (2024)
- [Calibrated Multi-Level Quantile Forecasting](https://mlsanity.com/paper/2512.23671/) (2025)
- [Hierarchies of Calibration: Classification meets Regression](https://mlsanity.com/paper/2606.03245/) (2026)
- [Calibration Bands for Mean Estimates within the Exponential Dispersion Family](https://mlsanity.com/paper/2503.18896/) (2025)
- [Recipes for Calibration Checks in Safety-Critical Applications](https://mlsanity.com/paper/2604.26479/) (2026)
- [Assessing model calibration with boosting trees](https://mlsanity.com/paper/2606.08084/) (2026)
- [Conditional expectation network for SHAP](https://mlsanity.com/paper/2307.10654/) (2023)
- [Simplifying Random Forests' Probabilistic Forecasts](https://mlsanity.com/paper/2408.12332/) (2024)
- [Calibrated Multivariate Regression with Localized PIT Mappings](https://mlsanity.com/paper/2409.10855/) (2024)
- [Variable transformations in consistent loss functions](https://mlsanity.com/paper/2502.16542/) (2025)
- [Explainable Boosting Machine for Predicting Claim Severity and Frequency in Car Insurance](https://mlsanity.com/paper/2503.21321/) (2025)
- [Adaptive Bayesian Very Short-Term Wind Power Forecasting Based on the Generalised Logit Transformation](https://mlsanity.com/paper/2505.06310/) (2025)
- [Residual Distribution Predictive Systems](https://mlsanity.com/paper/2510.26914/) (2025)
- [Nonparametric Distribution Regression Re-calibration](https://mlsanity.com/paper/2602.13362/) (2026)
- [CAIRO: Decoupling Order from Scale in Regression](https://mlsanity.com/paper/2602.14440/) (2026)
- [Goal-Oriented Lower-Tail Calibration of Gaussian Processes for Bayesian Optimization](https://mlsanity.com/paper/2605.20145/) (2026)

## Abstract (arXiv)

Model diagnostics and forecast evaluation are two sides of the same coin. A common principle is that fitted or predicted distributions ought to be calibrated or reliable, ideally in the sense of auto-calibration, where the outcome is a random draw from the posited distribution. For binary responses, this is the universal concept of reliability. For real-valued outcomes, a general theory of calibration has been elusive, despite a recent surge of interest in distributional regression and machine learning. We develop a framework rooted in probability theory, which gives rise to hierarchies of calibration, and applies to both predictive distributions and stand-alone point forecasts. In a nutshell, a prediction - distributional or single-valued - is conditionally T-calibrated if it can be taken at face value in terms of the functional T. Whenever T is defined via an identification function - as in the cases of threshold (non) exceedance probabilities, quantiles, expectiles, and moments - auto-calibration implies T-calibration. We introduce population versions of T-reliability diagrams and revisit a score decomposition into measures of miscalibration (MCB), discrimination (DSC), and uncertainty (UNC). In empirical settings, stable and efficient estimators of T-reliability diagrams and score components arise via nonparametric isotonic regression and the pool-adjacent-violators algorithm. For in-sample model diagnostics, we propose a universal coefficient of determination, R^ast = fracDSC-MCBUNC, that nests and reinterprets the classical R2 in least squares (mean) regression and its natural analogue R1 in quantile regression, yet applies to T-regression in general, with MCB ≥ 0, DSC ≥ 0, and R^ast ∈ [0,1] under modest conditions.