# Stratification of uncertainties recalibrated by isotonic regression and its impact on calibration error statistics

arXiv: https://arxiv.org/abs/2306.05180 · HTML: https://mlsanity.com/paper/2306.05180/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 15 references that are in the map.
- First posted on arXiv 8 Jun 2023.

## Builds on (most cited first, 8 of 8)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification](https://mlsanity.com/paper/2107.07511/) (2021)
- [Accurate Uncertainties for Deep Learning Using Calibrated Regression](https://mlsanity.com/paper/1807.00263/) (2018)
- [Evaluating and Calibrating Uncertainty Prediction in Regression Tasks](https://mlsanity.com/paper/1905.11659/) (2019)
- [Calibrated Uncertainty for Molecular Property Prediction using Ensembles of Message Passing Neural Networks](https://mlsanity.com/paper/2107.06068/) (2021)
- [Graph Neural Network Interatomic Potential Ensembles with Calibrated Aleatoric and Epistemic Uncertainty on Energy and Forces](https://mlsanity.com/paper/2305.16325/) (2023)
- [Properties of the ENCE and other MAD-based calibration metrics](https://mlsanity.com/paper/2305.11905/) (2023)
- [Centered Isotonic Regression: Point and Interval Estimation for Dose-Response Studies](https://mlsanity.com/paper/1701.05964/) (2017)

## Built on by (most cited first, 6 of 6)

- [Calibration in Machine Learning Uncertainty Quantification: beyond consistency to target adaptivity](https://mlsanity.com/paper/2309.06240/) (2023)
- [Can bin-wise scaling improve consistency and adaptivity of prediction uncertainty for machine learning regression ?](https://mlsanity.com/paper/2310.11978/) (2023)
- [Negative impact of heavy-tailed uncertainty and error distributions on the reliability of calibration statistics for machine learning regression tasks](https://mlsanity.com/paper/2402.10043/) (2024)
- [Validation of ML-UQ calibration statistics using simulated reference values: a sensitivity analysis](https://mlsanity.com/paper/2403.00423/) (2024)
- [The Calibrated Deepfake Trust Score (CDTS): Competence-Coupled Trust Degradation Across Deepfake Detectors](https://mlsanity.com/paper/2606.29484/) (2026)
- [ResLearn-XR: Residual Learning for Network Traffic and Quality-of-Experience-Aware Modeling in Extended Reality](https://mlsanity.com/paper/2609.04493/) (2026)

## Abstract (arXiv)

Abstract Post hoc recalibration of prediction uncertainties of machine learning regression problems by isotonic regression might present a problem for bin-based calibration error statistics (e.g. ENCE). Isotonic regression often produces stratified uncertainties, i.e. subsets of uncertainties with identical numerical values. Partitioning of the resulting data into equal-sized bins introduces an aleatoric component to the estimation of bin-based calibration statistics. The partitioning of stratified data into bins depends on the order of the data, which is typically an uncontrolled property of calibration test/validation sets. The tie-braking method of the ordering algorithm used for binning might also introduce an aleatoric component. I show on an example how this might significantly affect the calibration diagnostics.