mlsanityData to 7 Oct 2026

On Information-Theoretic Measures of Predictive Uncertainty

arXiv 2410.10786 ↗First posted 14 Oct 2024cs.LG cs.AI stat.MLcited by 16

Not on the map. Why?

Abstract (arXiv)

Reliable estimation of predictive uncertainty is crucial for machine learning applications, particularly in high-stakes scenarios where hedging against risks is essential. Despite its significance, there is no universal agreement on how to best quantify predictive uncertainty. In this work, we revisit core concepts to propose a framework for information-theoretic measures of predictive uncertainty. Our proposed framework categorizes predictive uncertainty measures according to two factors: (I) The predicting model (II) The approximation of the true predictive distribution. Examining all possible combinations of these two factors, we derive a set of predictive uncertainty measures that includes both known and newly introduced ones. We extensively evaluate these measures across a broad set of tasks, identifying conditions under which certain measures excel. Our findings show the importance of aligning the choice of uncertainty measure with the predicting model on in-distribution (ID) data, the limitations of epistemic uncertainty measures for out-of-distribution (OOD) data, and that the disentanglement between measures varies substantially between ID and OOD data. Together, these insights provide a more comprehensive understanding of predictive uncertainty measures, revealing their implicit assumptions and relationships.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 51 of its 81 references that are in the map. First posted on arXiv 14 Oct 2024.

Builds on

51 of 81 references.

References · 51

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  3. 2016Densely Connected Convolutional Networkscited by 6,691
  4. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  5. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  6. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  7. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  8. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  9. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
  10. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  11. 2015LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loopcited by 1,350
  12. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
38 more
  1. 2020Designing Network Design Spacescited by 954
  2. 2019Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methodscited by 903
  3. 2017Deep Bayesian Active Learning with Image Datacited by 812
  4. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  5. 2018Evidential Deep Learning to Quantify Classification Uncertaintycited by 728
  6. 2021A Survey of Uncertainty in Deep Neural Networkscited by 675
  7. 2018Predictive Uncertainty Estimation via Prior Networkscited by 555
  8. 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
  9. 2019Deep Ensembles: A Loss Landscape Perspectivecited by 438
  10. 2020Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awarenesscited by 389
  11. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 359
  12. 2019Deep Evidential Regressioncited by 354
  13. 2014Stochastic Gradient Hamiltonian Monte Carlocited by 341
  14. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 318
  15. 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 307
  16. 2019Cyclical Stochastic Gradient MCMC for Bayesian Deep Learningcited by 203
  17. 2021Deep Deterministic Uncertainty: A Simple Baselinecited by 200
  18. 2018Understanding Measures of Uncertainty for Adversarial Example Detectioncited by 177
  19. 2021On Hallucination and Predictive Uncertainty in Conditional Language Generationcited by 176
  20. 2022Quantifying Aleatoric and Epistemic Uncertainty in Machine Learning: Are Conditional Entropy and Mutual Information Appropriate Measures?cited by 124
  21. 2024To Believe or Not to Believe Your LLMcited by 120
  22. 2021DEUP: Direct Epistemic Uncertainty Predictioncited by 102
  23. 2022A Deeper Look into Aleatoric and Epistemic Uncertainty Disentanglementcited by 87
  24. 2024Benchmarking Uncertainty Disentanglement: Specialized Uncertainties for Specialized Taskscited by 78
  25. 2024Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AIcited by 60
  26. 2017Adversarial Examples, Uncertainty, and Transfer Testing Robustness in Gaussian Process Hybrid Deep Networkscited by 48
  27. 2023Is the Volume of a Credal Set a Good Measure for Epistemic Uncertainty?cited by 40
  28. 2022Nonparametric Uncertainty Quantification for Single Deterministic Neural Networkcited by 36
  29. 2023Second-Order Uncertainty Quantification: A Distance-Based Approachcited by 35
  30. 2024Improving Uncertainty Estimation through Semantically Diverse Language Generationcited by 34
  31. 2023Quantification of Uncertainty with Adversarial Modelscited by 28
  32. 2023Introducing an Improved Information-Theoretic Measure of Predictive Uncertaintycited by 28
  33. 2021Deep Ensembles from a Bayesian Perspectivecited by 27
  34. 2024From Risk to Uncertainty: Generating Predictive Uncertainty Measures via Bayesian Estimationcited by 27
  35. 2024Label-wise Aleatoric and Epistemic Uncertainty Quantificationcited by 22
  36. 2024Quantifying Aleatoric and Epistemic Uncertainty with Proper Scoring Rulescited by 21
  37. 2022Uncertainty Estimates of Predictions via a General Bias-Variance Decompositioncited by 20
  38. 2023Second-Order Uncertainty Quantification: Variance-Based Measurescited by 19

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 8; 2026: 6.

16 papers.

Sort:

By area:

  1. 2024Rethinking Aleatoric and Epistemic Uncertaintycited by 40
  2. 2025Position: Uncertainty Quantification Needs Reassessment for Large-language Model Agentscited by 27
  3. 2024Rethinking Uncertainty Estimation in LLMs: A Principled Single-Sequence Measurecited by 18
  4. 2025Uncertainty Quantification for Hallucination Detection in Large Language Models: Foundations, Methodology, and Future Directionscited by 10
  5. 2025Uncertainty as Feature Gaps: Epistemic Uncertainty Quantification of LLMs in Contextual Question-Answeringcited by 6
  6. 2025Uncertainty Quantification for Regression using Proper Scoring Rulescited by 5
  7. 2025Statistical Methods in Generative AIcited by 4
  8. 2026Semantic Uncertainty Quantification of Hallucinations in LLMs: A Quantum Tensor Network Based Methodcited by 4
  9. 2025Efficient semantic uncertainty quantification in language models via diversity-steered samplingcited by 2
  10. 2025Uncertainty Quantification with Proper Scoring Rules: Adjusting Measures to Prediction Taskscited by 1
  11. 2025Multidimensional Uncertainty Quantification via Optimal Transportcited by 1
  12. 2026Laws of Learning Dynamics and the Core of Learnerscited by 0
4 more
  1. 2026Deep Learning-Based Metamodeling of Nonlinear Stochastic Dynamic Systems under Parametric and Predictive Uncertaintycited by 0
  2. 2026On the QUEST for Uncertainty Quantification via Highest Density Regionscited by 0
  3. 2026Uncertainty Quantification for LLM Agents: A Taxonomy, an Evaluation Protocol, and an Empirical Studycited by 0
  4. 2026Beyond Answer Confidence: A Controlled Audit of Self-Knowledge in a Black-Box Decision Modelcited by 0