mlsanityData to 7 Oct 2026

Calibration in Machine Learning Uncertainty Quantification: beyond consistency to target adaptivity

arXiv 2309.06240 ↗First posted 12 Sep 2023stat.ML cs.LG physics.chem-phcited by 13

Not on the map. Why?

Abstract (arXiv)

Reliable uncertainty quantification (UQ) in machine learning (ML) regression tasks is becoming the focus of many studies in materials and chemical science. It is now well understood that average calibration is insufficient, and most studies implement additional methods testing the conditional calibration with respect to uncertainty, i.e. consistency. Consistency is assessed mostly by so-called reliability diagrams. There exists however another way beyond average calibration, which is conditional calibration with respect to input features, i.e. adaptivity. In practice, adaptivity is the main concern of the final users of a ML-UQ method, seeking for the reliability of predictions and uncertainties for any point in features space. This article aims to show that consistency and adaptivity are complementary validation targets, and that a good consistency does not imply a good adaptivity. Adapted validation methods are proposed and illustrated on a representative example.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 29 of its 61 references that are in the map. First posted on arXiv 12 Sep 2023.

Builds on

29 of 61 references.

References · 29

  1. 2017On Calibration of Modern Neural Networkscited by 4,860
  2. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  3. 2019Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methodscited by 903
  4. 2020A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challengescited by 813
  5. 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 777
  6. 2021A Survey of Uncertainty in Deep Neural Networkscited by 675
  7. 2018Accurate Uncertainties for Deep Learning Using Calibrated Regressioncited by 417
  8. 2018Uncertainty Estimates and Multi-Hypotheses Networks for Optical Flowcited by 135
  9. 2018High-Quality Prediction Intervals for Deep Learning: A Distribution-Free, Ensembled Approachcited by 117
  10. 2019Evaluating and Calibrating Uncertainty Prediction in Regression Taskscited by 85
  11. 2023A Survey on Uncertainty Quantification Methods for Deep Learningcited by 78
  12. 2020Beyond Pinball Loss: Quantile Methods for Calibrated Uncertainty Quantificationcited by 77
17 more
  1. 2020Uncertainty Quantification Using Neural Networks for Molecular Property Predictioncited by 66
  2. 2021Uncertainty Toolbox: an Open-Source Library for Assessing, Visualizing, and Improving Uncertainty Quantificationcited by 57
  3. 2021A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learningcited by 54
  4. 2020Individual Calibration with Randomized Forecastingcited by 50
  5. 2021Improving Conditional Coverage via Orthogonal Quantile Regressioncited by 48
  6. 2021Local Calibration: Metrics and Recalibrationcited by 23
  7. 2021Calibrated Uncertainty for Molecular Property Prediction using Ensembles of Message Passing Neural Networkscited by 21
  8. 2022Materials Property Prediction with Uncertainty Quantification: A Benchmark Studycited by 14
  9. 2021The Peril of Popular Deep Learning Uncertainty Estimation Methodscited by 11
  10. 2023Graph Neural Network Interatomic Potential Ensembles with Calibrated Aleatoric and Epistemic Uncertainty on Energy and Forcescited by 10
  11. 2020Metrics for Benchmarking and Uncertainty Quantification: Quality, Applicability, and a Path to Best Practices for Machine Learning in Chemistrycited by 9
  12. 2023Clarifying Trust of Materials Property Predictions using Neural Networks with Distribution-Specific Uncertainty Quantificationcited by 8
  13. 2023Properties of the ENCE and other MAD-based calibration metricscited by 7
  14. 2021Reliable Neural Networks for Regression Uncertainty Estimationcited by 6
  15. 2023Stratification of uncertainties recalibrated by isotonic regression and its impact on calibration error statisticscited by 6
  16. 2022Uncertainty quantification for predictions of atomistic neural networkscited by 5
  17. 2023Validation of uncertainty quantification metrics: a primer based on the consistency and adaptivity conceptscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 5; 2025: 4; 2026: 3.

13 papers.

Sort:

  1. 2024Uncertainty Quantification in Large Language Models Through Convex Hull Analysiscited by 12
  2. 2023Can bin-wise scaling improve consistency and adaptivity of prediction uncertainty for machine learning regression ?cited by 4
  3. 2025Trustworthy image-to-image translation: evaluating uncertainty calibration in unpaired training scenarioscited by 3
  4. 2025Uncertainty quantification with approximate variational learning for wearable photoplethysmography prediction taskscited by 3
  5. 2024Negative impact of heavy-tailed uncertainty and error distributions on the reliability of calibration statistics for machine learning regression taskscited by 1
  6. 2025Calibrated and uncertain? Evaluating uncertainty estimates in binary classification modelscited by 1
  7. 2025A systematic evaluation of uncertainty quantification techniques in deep learning: a case study in photoplethysmography signal analysiscited by 1
  8. 2024Validation of ML-UQ calibration statistics using simulated reference values: a sensitivity analysiscited by 0
  9. 2024On the good reliability of an interval-based metric to validate prediction uncertainty for machine learning regression taskscited by 0
  10. 2024Materials-Discovery Workflows Guided by Symbolic Regression: Identifying Acid-Stable Oxides for Electrocatalysiscited by 0
  11. 2026A Critical Examination of Active Learning Workflows in Materials Sciencecited by 0
  12. 2026How Understanding Forecast Uncertainty Resolves the Explainability Problem in Machine Learning Modelscited by 0
1 more
  1. 2026Trustworthy deep domain adaptation for wearable photoplethysmography signal analysis with decision-theoretic uncertainty quantificationcited by 0