mlsanityData to 14 Oct 2026

Stratification of uncertainties recalibrated by isotonic regression and its impact on calibration error statistics

arXiv 2306.05180 ↗First posted 8 Jun 2023stat.ME physics.chem-ph physics.data-ancited by 6

Not on the map. Why?

Abstract (arXiv)

Abstract Post hoc recalibration of prediction uncertainties of machine learning regression problems by isotonic regression might present a problem for bin-based calibration error statistics (e.g. ENCE). Isotonic regression often produces stratified uncertainties, i.e. subsets of uncertainties with identical numerical values. Partitioning of the resulting data into equal-sized bins introduces an aleatoric component to the estimation of bin-based calibration statistics. The partitioning of stratified data into bins depends on the order of the data, which is typically an uncontrolled property of calibration test/validation sets. The tie-braking method of the ordering algorithm used for binning might also introduce an aleatoric component. I show on an example how this might significantly affect the calibration diagnostics.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 8 of its 15 references that are in the map. First posted on arXiv 8 Jun 2023.

Builds on

8 of 15 references.

References · 8

  1. 2017On Calibration of Modern Neural Networkscited by 4,960
  2. 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 792
  3. 2018Accurate Uncertainties for Deep Learning Using Calibrated Regressioncited by 422
  4. 2019Evaluating and Calibrating Uncertainty Prediction in Regression Taskscited by 87
  5. 2021Calibrated Uncertainty for Molecular Property Prediction using Ensembles of Message Passing Neural Networkscited by 21
  6. 2023Graph Neural Network Interatomic Potential Ensembles with Calibrated Aleatoric and Epistemic Uncertainty on Energy and Forcescited by 10
  7. 2023Properties of the ENCE and other MAD-based calibration metricscited by 7
  8. 2017Centered Isotonic Regression: Point and Interval Estimation for Dose-Response Studiescited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 2; 2025: 0; 2026: 2.

6 papers.

  1. 2023Calibration in Machine Learning Uncertainty Quantification: beyond consistency to target adaptivitycited by 13
  2. 2023Can bin-wise scaling improve consistency and adaptivity of prediction uncertainty for machine learning regression ?cited by 4
  3. 2024Negative impact of heavy-tailed uncertainty and error distributions on the reliability of calibration statistics for machine learning regression taskscited by 1
  4. 2024Validation of ML-UQ calibration statistics using simulated reference values: a sensitivity analysiscited by 0
  5. 2026The Calibrated Deepfake Trust Score (CDTS): Competence-Coupled Trust Degradation Across Deepfake Detectorscited by 0
  6. 2026ResLearn-XR: Residual Learning for Network Traffic and Quality-of-Experience-Aware Modeling in Extended Realitycited by 0