mlsanityData to 7 Oct 2026

Validation of uncertainty quantification metrics: a primer based on the consistency and adaptivity concepts

arXiv 2303.07170 ↗First posted 13 Mar 2023physics.chem-ph physics.data-an stat.MLcited by 5

Not on the map. Why?

Abstract (arXiv)

The practice of uncertainty quantification (UQ) validation, notably in machine learning for the physico-chemical sciences, rests on several graphical methods (scattering plots, calibration curves, reliability diagrams and confidence curves) which explore complementary aspects of calibration, without covering all the desirable ones. For instance, none of these methods deals with the reliability of UQ metrics across the range of input features (adaptivity). Based on the complementary concepts of consistency and adaptivity, the toolbox of common validation methods for variance- and intervals- based UQ metrics is revisited with the aim to provide a better grasp on their capabilities. This study is conceived as an introduction to UQ validation, and all methods are derived from a few basic rules. The methods are illustrated and tested on synthetic datasets and representative examples extracted from the recent physico-chemical machine learning UQ literature.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 19 of its 46 references that are in the map. First posted on arXiv 13 Mar 2023.

Builds on

19 of 46 references.

References · 19

  1. 2017On Calibration of Modern Neural Networkscited by 4,860
  2. 2019Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methodscited by 903
  3. 2020A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challengescited by 813
  4. 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 777
  5. 2021A Survey of Uncertainty in Deep Neural Networkscited by 675
  6. 2019Measuring Calibration in Deep Learningcited by 443
  7. 2018Accurate Uncertainties for Deep Learning Using Calibrated Regressioncited by 417
  8. 2018Uncertainty Estimates and Multi-Hypotheses Networks for Optical Flowcited by 135
  9. 2018High-Quality Prediction Intervals for Deep Learning: A Distribution-Free, Ensembled Approachcited by 117
  10. 2019Evaluating and Calibrating Uncertainty Prediction in Regression Taskscited by 85
  11. 2023A Survey on Uncertainty Quantification Methods for Deep Learningcited by 78
  12. 2020Beyond Pinball Loss: Quantile Methods for Calibrated Uncertainty Quantificationcited by 77
7 more
  1. 2020Uncertainty Quantification Using Neural Networks for Molecular Property Predictioncited by 66
  2. 2021Improving Conditional Coverage via Orthogonal Quantile Regressioncited by 48
  3. 2021Local Calibration: Metrics and Recalibrationcited by 23
  4. 2021Calibrated Uncertainty for Molecular Property Prediction using Ensembles of Message Passing Neural Networkscited by 21
  5. 2022Materials Property Prediction with Uncertainty Quantification: A Benchmark Studycited by 14
  6. 2021The Peril of Popular Deep Learning Uncertainty Estimation Methodscited by 11
  7. 2022Uncertainty quantification for predictions of atomistic neural networkscited by 5

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 1; 2025: 1.

5 papers.

  1. 2023Calibration in Machine Learning Uncertainty Quantification: beyond consistency to target adaptivitycited by 13
  2. 2023Graph Neural Network Interatomic Potential Ensembles with Calibrated Aleatoric and Epistemic Uncertainty on Energy and Forcescited by 10
  3. 2023Can bin-wise scaling improve consistency and adaptivity of prediction uncertainty for machine learning regression ?cited by 4
  4. 2025Uncertainty quantification with approximate variational learning for wearable photoplethysmography prediction taskscited by 3
  5. 2024Negative impact of heavy-tailed uncertainty and error distributions on the reliability of calibration statistics for machine learning regression taskscited by 1