# Evaluating Epistemic Uncertainty: Beyond OOD Detection and Active Learning

arXiv: https://arxiv.org/abs/2607.14817 · HTML: https://mlsanity.com/paper/2607.14817/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 47 references that are in the map.
- First posted on arXiv 16 Jul 2026.

## Builds on (most cited first, 30 of 32)

- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods](https://mlsanity.com/paper/1910.09457/) (2019)
- [Selective Classification for Deep Neural Networks](https://mlsanity.com/paper/1705.08500/) (2017)
- [Evidential Deep Learning to Quantify Classification Uncertainty](https://mlsanity.com/paper/1806.01768/) (2018)
- [Human uncertainty makes classification more robust](https://mlsanity.com/paper/1908.07086/) (2019)
- [Deep Deterministic Uncertainty: A Simple Baseline](https://mlsanity.com/paper/2102.11582/) (2021)
- [OpenOOD v1.5: Enhanced Benchmark for Out-of-Distribution Detection](https://mlsanity.com/paper/2306.09301/) (2023)
- [Quantifying Aleatoric and Epistemic Uncertainty in Machine Learning: Are Conditional Entropy and Mutual Information Appropriate Measures?](https://mlsanity.com/paper/2209.03302/) (2022)
- [Bias-Reduced Uncertainty Estimation for Deep Neural Classifiers](https://mlsanity.com/paper/1805.08206/) (2018)
- [Machine Learning with a Reject Option: A survey](https://mlsanity.com/paper/2107.11277/) (2021)
- [DEUP: Direct Epistemic Uncertainty Prediction](https://mlsanity.com/paper/2102.08501/) (2021)
- [Benchmarking Uncertainty Disentanglement: Specialized Uncertainties for Specialized Tasks](https://mlsanity.com/paper/2402.19460/) (2024)
- [Optimal strategies for reject option classifiers](https://mlsanity.com/paper/2101.12523/) (2021)
- [Is one annotation enough? A data-centric image classification benchmark for noisy and ambiguous label estimation](https://mlsanity.com/paper/2207.06214/) (2022)
- [MiVOLO: Multi-input Transformer for Age and Gender Estimation](https://mlsanity.com/paper/2307.04616/) (2023)
- [On Second-Order Scoring Rules for Epistemic Uncertainty Quantification](https://mlsanity.com/paper/2301.12736/) (2023)
- [Overcoming Common Flaws in the Evaluation of Selective Classification Systems](https://mlsanity.com/paper/2407.01032/) (2024)
- [From Risk to Uncertainty: Generating Predictive Uncertainty Measures via Bayesian Estimation](https://mlsanity.com/paper/2402.10727/) (2024)
- [Augmenting Softmax Information for Selective Classification with Out-of-Distribution Data](https://mlsanity.com/paper/2207.07506/) (2022)
- [Quantifying Aleatoric and Epistemic Uncertainty with Proper Scoring Rules](https://mlsanity.com/paper/2404.12215/) (2024)
- [Uncertainty Estimates of Predictions via a General Bias-Variance Decomposition](https://mlsanity.com/paper/2210.12256/) (2022)
- [Investigating Bias in Deep Face Analysis: The KANFace Dataset and Empirical Study](https://mlsanity.com/paper/2005.07302/) (2020)
- [Integrating Uncertainty Awareness into Conformalized Quantile Regression](https://mlsanity.com/paper/2306.08693/) (2023)
- [How to Fix a Broken Confidence Estimator: Evaluating Post-hoc Methods for Selective Classification with Deep Neural Networks](https://mlsanity.com/paper/2305.15508/) (2023)
- [Position: Epistemic uncertainty estimation methods are fundamentally incomplete](https://mlsanity.com/paper/2505.23506/) (2025)
- [Measuring Orthogonality as the Blind-Spot of Uncertainty Disentanglement](https://mlsanity.com/paper/2408.12175/) (2024)
- [Uncertainty Quantification for Machine Learning: One Size Does Not Fit All](https://mlsanity.com/paper/2512.12341/) (2025)
- [SCOD: From Heuristics to Theory](https://mlsanity.com/paper/2403.16916/) (2024)

## Built on by (most cited first, 2 of 2)

- [A Ground-Truth Framework for Uncertainty Disentanglement with Posterior Risk](https://mlsanity.com/paper/2608.05995/) (2026)
- [Rethinking Epistemic Uncertainty in Node Classification through Information Growth](https://mlsanity.com/paper/2610.03418/) (2026)

## Abstract (arXiv)

Current evaluation of epistemic uncertainty relies on tasks such as out-ofdistribution detection and active learning. However, the Bayes-optimal decision strategies for these tasks do not coincide with the scores commonly used to quantify epistemic uncertainty. Building on the epistemic reject-option framework, we evaluate epistemic uncertainty using its ability to identify regret, the reducible error. Formulating selective prediction as a constrained optimization over coverage, expected risk, and regret, we prove the optimal selector is a thresholded convex combination of the ground-truth aleatoric and epistemic uncertainties. This theoretical unification exposes a weakness in recent uncertainty disentanglement literature: we demonstrate that standard correlation metrics between learned components do not necessarily predict their actual operational utility. We instead propose to evaluate the achievable risk, regret, coverage surface of the decomposition as a diagnostic for joint disentanglement and utility. Benchmarking standard methods on datasets with dense human annotations reveals that decision-theoretic rankings can disagree substantially with proxy-task rankings, including pairwise rank inversions between methods that are top-ranked on one criterion and bottom-ranked on other.