# A Unified Evaluation Framework for Epistemic Predictions

arXiv: https://arxiv.org/abs/2501.16912 · HTML: https://mlsanity.com/paper/2501.16912/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 87 references that are in the map.
- First posted on arXiv 28 Jan 2025.

## Builds on (most cited first, 16 of 16)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods](https://mlsanity.com/paper/1910.09457/) (2019)
- [Evidential Deep Learning to Quantify Classification Uncertainty](https://mlsanity.com/paper/1806.01768/) (2018)
- [Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Users](https://mlsanity.com/paper/2007.06823/) (2020)
- [Deep Deterministic Uncertainty: A Simple Baseline](https://mlsanity.com/paper/2102.11582/) (2021)
- [Bayesian Neural Networks: An Introduction and Survey](https://mlsanity.com/paper/2006.12024/) (2020)
- [Credal Bayesian Deep Learning](https://mlsanity.com/paper/2302.09656/) (2023)
- [Fast Predictive Uncertainty for Classification with Bayesian Deep Networks](https://mlsanity.com/paper/2003.01227/) (2020)
- [Credal Learning Theory](https://mlsanity.com/paper/2402.00957/) (2024)
- [CreINNs: Credal-Set Interval Neural Networks for Uncertainty Estimation in Classification Tasks](https://mlsanity.com/paper/2401.05043/) (2024)
- [Random-Set Neural Networks (RS-NN)](https://mlsanity.com/paper/2307.05772/) (2023)
- [An evidential classifier based on Dempster-Shafer theory and deep learning](https://mlsanity.com/paper/2103.13549/) (2021)
- [Ensemble Quantile Networks: Uncertainty-Aware Reinforcement Learning with Applications in Autonomous Driving](https://mlsanity.com/paper/2105.10266/) (2021)
- [Epistemic Deep Learning](https://mlsanity.com/paper/2206.07609/) (2022)

## Built on by (most cited first, 10 of 10)

- [Epistemic Artificial Intelligence is Essential for Machine Learning Models to Truly 'Know When They Do Not Know'](https://mlsanity.com/paper/2505.04950/) (2025)
- [Learning Credal Ensembles via Distributionally Robust Optimization](https://mlsanity.com/paper/2602.08470/) (2026)
- [Epistemic Wrapping for Uncertainty Quantification](https://mlsanity.com/paper/2505.02277/) (2025)
- [Random-Set Graph Neural Networks](https://mlsanity.com/paper/2605.11987/) (2026)
- [Statistical inference with belief functions: A survey](https://mlsanity.com/paper/2605.07908/) (2026)
- [Credal Large Language Models for Semantic Commitment under Uncertainty](https://mlsanity.com/paper/2608.23244/) (2026)
- [Set-based v.s. Distribution-based Representations of Epistemic Uncertainty: A Comparative Study](https://mlsanity.com/paper/2602.22747/) (2026)
- [A formal definition and meta-model for a machine theory of mind](https://mlsanity.com/paper/2606.03471/) (2026)
- [What Intermediate Layers Know: Detecting Jailbreaks from Entropy Dynamics](https://mlsanity.com/paper/2606.25182/) (2026)
- [XU-RS: Explaining Credal Width in Random-Set Language Models](https://mlsanity.com/paper/2609.37594/) (2026)

## Abstract (arXiv)

Predictions of uncertainty-aware models are diverse, ranging from single point estimates (often averaged over prediction samples) to predictive distributions, to set-valued or credal-set representations. We propose a novel unified evaluation framework for uncertainty-aware classifiers, applicable to a wide range of model classes, which allows users to tailor the trade-off between accuracy and precision of predictions via a suitably designed performance metric. This makes possible the selection of the most suitable model for a particular real-world application as a function of the desired trade-off. Our experiments, concerning Bayesian, ensemble, evidential, deterministic, credal and belief function classifiers on the CIFAR-10, MNIST and CIFAR-100 datasets, show that the metric behaves as desired.