# In-sample calibration yields conformal calibration guarantees

arXiv: https://arxiv.org/abs/2503.03841 · HTML: https://mlsanity.com/paper/2503.03841/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Mar 2025.

## Built on by (most cited first, 10 of 10)

- [Enforcing tail calibration when training probabilistic forecast models](https://mlsanity.com/paper/2506.13687/) (2025)
- [Distributional Regression with Tabular Foundation Models: Evaluating Probabilistic Predictions via Proper Scoring Rules](https://mlsanity.com/paper/2603.08206/) (2026)
- [A Gentle Introduction to Conformal Time Series Forecasting](https://mlsanity.com/paper/2511.13608/) (2025)
- [CRPS-Optimal Binning for Univariate Conformal Regression](https://mlsanity.com/paper/2603.22000/) (2026)
- [Hierarchies of Calibration: Classification meets Regression](https://mlsanity.com/paper/2606.03245/) (2026)
- [Residual Distribution Predictive Systems](https://mlsanity.com/paper/2510.26914/) (2025)
- [Inductive Venn-Abers and related regressors](https://mlsanity.com/paper/2605.06646/) (2026)
- [Generalized Conformal Predictive Systems Under Distributional Shifts](https://mlsanity.com/paper/2606.11044/) (2026)
- [An analysis of binary isotonic regression: degrees of freedom and implications for calibration](https://mlsanity.com/paper/2607.27301/) (2026)
- [Towards Calibrated Probabilistic Forecasts for Events of Interest via Outcome-Conditional Recalibration](https://mlsanity.com/paper/2610.10076/) (2026)

## Abstract (arXiv)

Conformal prediction produces a set of predictions that has out-of-sample calibration guarantees by construction, under the assumption of exchangeability. In this work, we study the calibration properties of conformal predictive systems, which issue sets of predictive distributions for real-valued outcomes. We demonstrate that conformal predictive systems implicitly exploit prediction methods that are in-sample calibrated to construct sets that are guaranteed to contain a calibrated predictive distribution out-of-sample. This allows us to take any prediction method that is in-sample calibrated, and conformalize it to obtain a predictive system with out-of-sample calibration guarantees. While the satisfied notion of calibration is typically that prediction intervals derived from the predictive distribution have the correct marginal coverage, we show that this line of reasoning can be extended to stronger conditional notions of calibration that are common in statistical forecasting theory. Using this, we introduce two predictive systems that satisfy stronger out-of-sample calibration guarantees than existing conformal predictive systems. The first method corresponds to a binning of the data, while the second leverages isotonic distributional regression (IDR), a non-parametric distributional regression method under order constraints. We study the theoretical properties of these new predictive systems, and compare their performance in a simulation experiment. They are then applied to two case studies on European temperature forecasts and on predictions for the length of patient stay in Swiss intensive care units. Both approaches are found to outperform existing conformal predictive systems, while conformal IDR additionally provides a natural method for quantifying epistemic uncertainty of the predictions.