mlsanityData to 7 Oct 2026

Introducing an Improved Information-Theoretic Measure of Predictive Uncertainty

arXiv 2311.08309 ↗First posted 14 Nov 2023cs.LG stat.MLcited by 28

Not on the map. Why?

Abstract (arXiv)

Applying a machine learning model for decision-making in the real world requires to distinguish what the model knows from what it does not. A critical factor in assessing the knowledge of a model is to quantify its predictive uncertainty. Predictive uncertainty is commonly measured by the entropy of the Bayesian model average (BMA) predictive distribution. Yet, the properness of this current measure of predictive uncertainty was recently questioned. We provide new insights regarding those limitations. Our analyses show that the current measure erroneously assumes that the BMA predictive distribution is equivalent to the predictive distribution of the true model that generated the dataset. Consequently, we introduce a theoretically grounded measure to overcome these limitations. We experimentally verify the benefits of our introduced measure of predictive uncertainty. We find that our introduced measure behaves more reasonably in controlled synthetic tasks. Moreover, our evaluations on ImageNet demonstrate that our introduced measure is advantageous in real-world applications utilizing predictive uncertainty.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 24 of its 43 references that are in the map. First posted on arXiv 14 Nov 2023.

Builds on

24 of 43 references.

References · 24

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  3. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  4. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  5. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  6. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
  7. 2019Natural Adversarial Examplescited by 1,401
  8. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  9. 2019Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methodscited by 903
  10. 2018Predictive Uncertainty Estimation via Prior Networkscited by 555
  11. 2017Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methodscited by 542
  12. 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
12 more
  1. 2020Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awarenesscited by 389
  2. 2019Cyclical Stochastic Gradient MCMC for Bayesian Deep Learningcited by 203
  3. 2021Deep Deterministic Uncertainty: A Simple Baselinecited by 200
  4. 2018Understanding Measures of Uncertainty for Adversarial Example Detectioncited by 177
  5. 2022Quantifying Aleatoric and Epistemic Uncertainty in Machine Learning: Are Conditional Entropy and Mutual Information Appropriate Measures?cited by 124
  6. 2021On Feature Collapse and Deep Kernel Learning for Single Forward Pass Uncertaintycited by 87
  7. 2019A Systematic Comparison of Bayesian Deep Learning Robustness in Diabetic Retinopathy Taskscited by 62
  8. 2021On the Practicality of Deterministic Epistemic Uncertaintycited by 62
  9. 2022Benchmarking Bayesian Deep Learning on Diabetic Retinopathy Detection Taskscited by 49
  10. 2017Adversarial Examples, Uncertainty, and Transfer Testing Robustness in Gaussian Process Hybrid Deep Networkscited by 48
  11. 2023Is the Volume of a Credal Set a Good Measure for Epistemic Uncertainty?cited by 40
  12. 2023Quantification of Uncertainty with Adversarial Modelscited by 28

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 11; 2025: 6; 2026: 10.

28 papers.

Sort:

By area:

  1. 2024Rethinking Aleatoric and Epistemic Uncertaintycited by 40
  2. 2023Second-Order Uncertainty Quantification: A Distance-Based Approachcited by 35
  3. 2024Is Epistemic Uncertainty Faithfully Represented by Evidential Deep Learning Methods?cited by 34
  4. 2024Improving Uncertainty Estimation through Semantically Diverse Language Generationcited by 34
  5. 2024From Risk to Uncertainty: Generating Predictive Uncertainty Measures via Bayesian Estimationcited by 27
  6. 2024Quantifying Aleatoric and Epistemic Uncertainty with Proper Scoring Rulescited by 21
  7. 2024Rethinking Uncertainty Estimation in LLMs: A Principled Single-Sequence Measurecited by 18
  8. 2024On Information-Theoretic Measures of Predictive Uncertaintycited by 16
  9. 2024Enhancing Vision-Language Model Reliability with Uncertainty-Guided Dropout Decodingcited by 14
  10. 2026Complementing Self-Consistency with Cross-Model Disagreement for Uncertainty Quantificationcited by 9
  11. 2024Provable Uncertainty Decomposition via Higher-Order Calibrationcited by 8
  12. 2025Reducing Aleatoric and Epistemic Uncertainty through Multi-modal Data Acquisitioncited by 5
16 more
  1. 2024Measuring Orthogonality as the Blind-Spot of Uncertainty Disentanglementcited by 4
  2. 2025Uncertainty Estimation using Variance-Gated Distributionscited by 2
  3. 2025ESI: Epistemic Uncertainty Quantification via Semantic-preserving Intervention for Large Language Modelscited by 2
  4. 2025Uncertainty Quantification for Regression: A Unified Framework based on kernel scorescited by 2
  5. 2026Variance-Gated Ensembles: An Epistemic-Aware Framework for Uncertainty Estimationcited by 2
  6. 2024Navigating Uncertainty in Medical Image Segmentationcited by 1
  7. 2025Multidimensional Uncertainty Quantification via Optimal Transportcited by 1
  8. 2025Deep Ensembles for Epistemic Uncertainty: A Frequentist Perspectivecited by 1
  9. 2026SphUnc: Hyperspherical Uncertainty Decomposition and Causal Identification via Information Geometrycited by 1
  10. 2026Rethinking Uncertainty Quantification and Entanglement in Image Segmentationcited by 1
  11. 2026Subjective Risk Decomposition: A New View for Uncertainty Quantificationcited by 1
  12. 2026Improving Detection of Rare Nodes in Hierarchical Multi-Label Learningcited by 0
  13. 2026On the QUEST for Uncertainty Quantification via Highest Density Regionscited by 0
  14. 2026Uncertainty quantification for trustworthy deep learning: Methods and measurescited by 0
  15. 2026Flexible Spectral-Normalized Neural Gaussian Process for Dynamic Aperture Predictioncited by 0
  16. 2026Structural Limits of the Information-Theoretic Uncertainty Decompositioncited by 0