# Quantifying Aleatoric and Epistemic Uncertainty with Proper Scoring Rules

arXiv: https://arxiv.org/abs/2404.12215 · HTML: https://mlsanity.com/paper/2404.12215/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 33 references that are in the map.
- First posted on arXiv 18 Apr 2024.

## Builds on (most cited first, 10 of 10)

- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods](https://mlsanity.com/paper/1910.09457/) (2019)
- [Understanding Measures of Uncertainty for Adversarial Example Detection](https://mlsanity.com/paper/1803.08533/) (2018)
- [Quantifying Aleatoric and Epistemic Uncertainty in Machine Learning: Are Conditional Entropy and Mutual Information Appropriate Measures?](https://mlsanity.com/paper/2209.03302/) (2022)
- [DropConnect Is Effective in Modeling Uncertainty of Bayesian Deep Networks](https://mlsanity.com/paper/1906.04569/) (2019)
- [Pitfalls of Epistemic Uncertainty Quantification through Loss Minimisation](https://mlsanity.com/paper/2203.06102/) (2022)
- [On Uncertainty, Tempering, and Data Augmentation in Bayesian Classification](https://mlsanity.com/paper/2203.16481/) (2022)
- [Introducing an Improved Information-Theoretic Measure of Predictive Uncertainty](https://mlsanity.com/paper/2311.08309/) (2023)
- [From Risk to Uncertainty: Generating Predictive Uncertainty Measures via Bayesian Estimation](https://mlsanity.com/paper/2402.10727/) (2024)
- [Second-Order Uncertainty Quantification: Variance-Based Measures](https://mlsanity.com/paper/2401.00276/) (2023)

## Built on by (most cited first, 21 of 21)

- [Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph](https://mlsanity.com/paper/2406.15627/) (2024)
- [Rethinking Aleatoric and Epistemic Uncertainty](https://mlsanity.com/paper/2412.20892/) (2024)
- [Rethinking Uncertainty Estimation in LLMs: A Principled Single-Sequence Measure](https://mlsanity.com/paper/2412.15176/) (2024)
- [On Information-Theoretic Measures of Predictive Uncertainty](https://mlsanity.com/paper/2410.10786/) (2024)
- [Scalable Bayesian Learning with posteriors](https://mlsanity.com/paper/2406.00104/) (2024)
- [Provable Uncertainty Decomposition via Higher-Order Calibration](https://mlsanity.com/paper/2412.18808/) (2024)
- [Position: Epistemic uncertainty estimation methods are fundamentally incomplete](https://mlsanity.com/paper/2505.23506/) (2025)
- [CLEAR: Calibrated Learning for Epistemic and Aleatoric Risk](https://mlsanity.com/paper/2507.08150/) (2025)
- [Hallucination Detection on a Budget: Efficient Bayesian Estimation of Semantic Entropy](https://mlsanity.com/paper/2504.03579/) (2025)
- [Uncertainty Quantification for Regression using Proper Scoring Rules](https://mlsanity.com/paper/2509.26610/) (2025)
- [Uncertainty Quantification for Machine Learning: One Size Does Not Fit All](https://mlsanity.com/paper/2512.12341/) (2025)
- [Uncertainty Quantification for Regression: A Unified Framework based on kernel scores](https://mlsanity.com/paper/2510.25599/) (2025)
- [Quantification of Credal Uncertainty: A Distance-Based Approach](https://mlsanity.com/paper/2603.27270/) (2026)
- [Evaluating Epistemic Uncertainty: Beyond OOD Detection and Active Learning](https://mlsanity.com/paper/2607.14817/) (2026)
- [Multidimensional Uncertainty Quantification via Optimal Transport](https://mlsanity.com/paper/2509.22380/) (2025)
- [Epistemic Reject Option Prediction](https://mlsanity.com/paper/2511.04855/) (2025)
- [Subjective Risk Decomposition: A New View for Uncertainty Quantification](https://mlsanity.com/paper/2607.15196/) (2026)
- [Beyond Accuracy: Evaluating Posterior Fidelity of Diffusion Inverse Solvers](https://mlsanity.com/paper/2602.04189/) (2026)
- [The Minimax Rate of Perturbed Second-Order Calibration](https://mlsanity.com/paper/2605.07808/) (2026)
- [The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing](https://mlsanity.com/paper/2606.25108/) (2026)
- [How Much Imprecision is Enough Imprecision in my Classifier? A Practical Elicitation Procedure](https://mlsanity.com/paper/2609.33352/) (2026)

## Abstract (arXiv)

Uncertainty representation and quantification are paramount in machine learning and constitute an important prerequisite for safety-critical applications. In this paper, we propose novel measures for the quantification of aleatoric and epistemic uncertainty based on proper scoring rules, which are loss functions with the meaningful property that they incentivize the learner to predict ground-truth (conditional) probabilities. We assume two common representations of (epistemic) uncertainty, namely, in terms of a credal set, i.e. a set of probability distributions, or a second-order distribution, i.e., a distribution over probability distributions. Our framework establishes a natural bridge between these representations. We provide a formal justification of our approach and introduce new measures of epistemic and aleatoric uncertainty as concrete instantiations.