# Calibration in Machine Learning Uncertainty Quantification: beyond consistency to target adaptivity

arXiv: https://arxiv.org/abs/2309.06240 · HTML: https://mlsanity.com/paper/2309.06240/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 61 references that are in the map.
- First posted on arXiv 12 Sep 2023.

## Builds on (most cited first, 29 of 29)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods](https://mlsanity.com/paper/1910.09457/) (2019)
- [A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges](https://mlsanity.com/paper/2011.06225/) (2020)
- [A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification](https://mlsanity.com/paper/2107.07511/) (2021)
- [A Survey of Uncertainty in Deep Neural Networks](https://mlsanity.com/paper/2107.03342/) (2021)
- [Accurate Uncertainties for Deep Learning Using Calibrated Regression](https://mlsanity.com/paper/1807.00263/) (2018)
- [Uncertainty Estimates and Multi-Hypotheses Networks for Optical Flow](https://mlsanity.com/paper/1802.07095/) (2018)
- [High-Quality Prediction Intervals for Deep Learning: A Distribution-Free, Ensembled Approach](https://mlsanity.com/paper/1802.07167/) (2018)
- [Evaluating and Calibrating Uncertainty Prediction in Regression Tasks](https://mlsanity.com/paper/1905.11659/) (2019)
- [A Survey on Uncertainty Quantification Methods for Deep Learning](https://mlsanity.com/paper/2302.13425/) (2023)
- [Beyond Pinball Loss: Quantile Methods for Calibrated Uncertainty Quantification](https://mlsanity.com/paper/2011.09588/) (2020)
- [Uncertainty Quantification Using Neural Networks for Molecular Property Prediction](https://mlsanity.com/paper/2005.10036/) (2020)
- [A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learning](https://mlsanity.com/paper/2109.02355/) (2021)
- [Uncertainty Toolbox: an Open-Source Library for Assessing, Visualizing, and Improving Uncertainty Quantification](https://mlsanity.com/paper/2109.10254/) (2021)
- [Individual Calibration with Randomized Forecasting](https://mlsanity.com/paper/2006.10288/) (2020)
- [Improving Conditional Coverage via Orthogonal Quantile Regression](https://mlsanity.com/paper/2106.00394/) (2021)
- [Local Calibration: Metrics and Recalibration](https://mlsanity.com/paper/2102.10809/) (2021)
- [Calibrated Uncertainty for Molecular Property Prediction using Ensembles of Message Passing Neural Networks](https://mlsanity.com/paper/2107.06068/) (2021)
- [Materials Property Prediction with Uncertainty Quantification: A Benchmark Study](https://mlsanity.com/paper/2211.02235/) (2022)
- [The Peril of Popular Deep Learning Uncertainty Estimation Methods](https://mlsanity.com/paper/2112.05000/) (2021)
- [Graph Neural Network Interatomic Potential Ensembles with Calibrated Aleatoric and Epistemic Uncertainty on Energy and Forces](https://mlsanity.com/paper/2305.16325/) (2023)
- [Metrics for Benchmarking and Uncertainty Quantification: Quality, Applicability, and a Path to Best Practices for Machine Learning in Chemistry](https://mlsanity.com/paper/2010.00110/) (2020)
- [Clarifying Trust of Materials Property Predictions using Neural Networks with Distribution-Specific Uncertainty Quantification](https://mlsanity.com/paper/2302.02595/) (2023)
- [Properties of the ENCE and other MAD-based calibration metrics](https://mlsanity.com/paper/2305.11905/) (2023)
- [Reliable Neural Networks for Regression Uncertainty Estimation](https://mlsanity.com/paper/2109.08213/) (2021)
- [Stratification of uncertainties recalibrated by isotonic regression and its impact on calibration error statistics](https://mlsanity.com/paper/2306.05180/) (2023)
- [Uncertainty quantification for predictions of atomistic neural networks](https://mlsanity.com/paper/2207.06916/) (2022)
- [Validation of uncertainty quantification metrics: a primer based on the consistency and adaptivity concepts](https://mlsanity.com/paper/2303.07170/) (2023)

## Built on by (most cited first, 13 of 13)

- [Uncertainty Quantification in Large Language Models Through Convex Hull Analysis](https://mlsanity.com/paper/2406.19712/) (2024)
- [Can bin-wise scaling improve consistency and adaptivity of prediction uncertainty for machine learning regression ?](https://mlsanity.com/paper/2310.11978/) (2023)
- [Trustworthy image-to-image translation: evaluating uncertainty calibration in unpaired training scenarios](https://mlsanity.com/paper/2501.17570/) (2025)
- [Uncertainty quantification with approximate variational learning for wearable photoplethysmography prediction tasks](https://mlsanity.com/paper/2505.11412/) (2025)
- [Negative impact of heavy-tailed uncertainty and error distributions on the reliability of calibration statistics for machine learning regression tasks](https://mlsanity.com/paper/2402.10043/) (2024)
- [Calibrated and uncertain? Evaluating uncertainty estimates in binary classification models](https://mlsanity.com/paper/2508.11460/) (2025)
- [A systematic evaluation of uncertainty quantification techniques in deep learning: a case study in photoplethysmography signal analysis](https://mlsanity.com/paper/2511.00301/) (2025)
- [Validation of ML-UQ calibration statistics using simulated reference values: a sensitivity analysis](https://mlsanity.com/paper/2403.00423/) (2024)
- [On the good reliability of an interval-based metric to validate prediction uncertainty for machine learning regression tasks](https://mlsanity.com/paper/2408.13089/) (2024)
- [Materials-Discovery Workflows Guided by Symbolic Regression: Identifying Acid-Stable Oxides for Electrocatalysis](https://mlsanity.com/paper/2412.05947/) (2024)
- [A Critical Examination of Active Learning Workflows in Materials Science](https://mlsanity.com/paper/2601.05946/) (2026)
- [How Understanding Forecast Uncertainty Resolves the Explainability Problem in Machine Learning Models](https://mlsanity.com/paper/2602.00179/) (2026)
- [Trustworthy deep domain adaptation for wearable photoplethysmography signal analysis with decision-theoretic uncertainty quantification](https://mlsanity.com/paper/2604.17480/) (2026)

## Abstract (arXiv)

Reliable uncertainty quantification (UQ) in machine learning (ML) regression tasks is becoming the focus of many studies in materials and chemical science. It is now well understood that average calibration is insufficient, and most studies implement additional methods testing the conditional calibration with respect to uncertainty, i.e. consistency. Consistency is assessed mostly by so-called reliability diagrams. There exists however another way beyond average calibration, which is conditional calibration with respect to input features, i.e. adaptivity. In practice, adaptivity is the main concern of the final users of a ML-UQ method, seeking for the reliability of predictions and uncertainties for any point in features space. This article aims to show that consistency and adaptivity are complementary validation targets, and that a good consistency does not imply a good adaptivity. Adapted validation methods are proposed and illustrated on a representative example.