# Benchmarking Uncertainty Disentanglement: Specialized Uncertainties for Specialized Tasks

arXiv: https://arxiv.org/abs/2402.19460 · HTML: https://mlsanity.com/paper/2402.19460/

## Key facts

- Cited by 78 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 47 references that are in the map.
- First posted on arXiv 29 Feb 2024.

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](https://mlsanity.com/paper/1807.03888/) (2018)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness](https://mlsanity.com/paper/2006.10108/) (2020)
- [Laplace Redux -- Effortless Bayesian Deep Learning](https://mlsanity.com/paper/2106.14806/) (2021)
- [Learning Loss for Active Learning](https://mlsanity.com/paper/1905.03677/) (2019)
- [Are we done with ImageNet?](https://mlsanity.com/paper/2006.07159/) (2020)
- [Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations](https://mlsanity.com/paper/2110.12088/) (2021)
- [Human uncertainty makes classification more robust](https://mlsanity.com/paper/1908.07086/) (2019)
- [Deep Deterministic Uncertainty: A Simple Baseline](https://mlsanity.com/paper/2102.11582/) (2021)
- [Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks](https://mlsanity.com/paper/1511.06314/) (2015)
- [Quantifying Aleatoric and Epistemic Uncertainty in Machine Learning: Are Conditional Entropy and Mutual Information Appropriate Measures?](https://mlsanity.com/paper/2209.03302/) (2022)
- [Plex: Towards Reliability using Pretrained Large Model Extensions](https://mlsanity.com/paper/2207.07411/) (2022)
- [A Deeper Look into Aleatoric and Epistemic Uncertainty Disentanglement](https://mlsanity.com/paper/2204.09308/) (2022)
- [Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learning](https://mlsanity.com/paper/2106.04015/) (2021)
- [On the Practicality of Deterministic Epistemic Uncertainty](https://mlsanity.com/paper/2107.00649/) (2021)
- [Eliciting and Learning with Soft Labels from Every Annotator](https://mlsanity.com/paper/2207.00810/) (2022)
- [On Second-Order Scoring Rules for Epistemic Uncertainty Quantification](https://mlsanity.com/paper/2301.12736/) (2023)
- [What Can We Learn From The Selective Prediction And Uncertainty Estimation Performance Of 523 Imagenet Classifiers](https://mlsanity.com/paper/2302.11874/) (2023)
- [A framework for benchmarking class-out-of-distribution detection and its application to ImageNet](https://mlsanity.com/paper/2302.11893/) (2023)
- [Probabilistic Contrastive Learning Recovers the Correct Aleatoric Uncertainty of Ambiguous Inputs](https://mlsanity.com/paper/2302.02865/) (2023)
- [Sources of Uncertainty in Supervised Machine Learning -- A Statisticians' View](https://mlsanity.com/paper/2305.16703/) (2023)
- [Uncertainty Estimates of Predictions via a General Bias-Variance Decomposition](https://mlsanity.com/paper/2210.12256/) (2022)
- [URL: A Representation Learning Benchmark for Transferable Uncertainty Estimates](https://mlsanity.com/paper/2307.03810/) (2023)
- [Ensemble-based Uncertainty Quantification: Bayesian versus Credal Inference](https://mlsanity.com/paper/2107.10384/) (2021)

## Built on by (most cited first, 30 of 78)

- [Rethinking Aleatoric and Epistemic Uncertainty](https://mlsanity.com/paper/2412.20892/) (2024)
- [Position: Uncertainty Quantification Needs Reassessment for Large-language Model Agents](https://mlsanity.com/paper/2505.22655/) (2025)
- [Revisiting Uncertainty Estimation and Calibration of Large Language Models](https://mlsanity.com/paper/2505.23854/) (2025)
- [Conformalized Credal Regions for Classification with Ambiguous Ground Truth](https://mlsanity.com/paper/2411.04852/) (2024)
- [On Information-Theoretic Measures of Predictive Uncertainty](https://mlsanity.com/paper/2410.10786/) (2024)
- [Query-Level Uncertainty in Large Language Models](https://mlsanity.com/paper/2506.09669/) (2025)
- [Anthropomimetic Uncertainty: What Verbalized Uncertainty in Language Models is Missing](https://mlsanity.com/paper/2507.10587/) (2025)
- [What Variables Affect Out-of-Distribution Generalization in Pretrained Models?](https://mlsanity.com/paper/2405.15018/) (2024)
- [Out-of-Distribution Segmentation in Autonomous Driving: Problems and State of the Art](https://mlsanity.com/paper/2503.08695/) (2025)
- [Are We Done with Object-Centric Learning?](https://mlsanity.com/paper/2504.07092/) (2025)
- [A Structured Review of Literature on Uncertainty in Machine Learning & Deep Learning](https://mlsanity.com/paper/2406.00332/) (2024)
- [Fine-Grained Uncertainty Decomposition in Large Language Models: A Spectral Approach](https://mlsanity.com/paper/2509.22272/) (2025)
- [Uncertainty Estimation by Flexible Evidential Deep Learning](https://mlsanity.com/paper/2510.18322/) (2025)
- [Position: Epistemic uncertainty estimation methods are fundamentally incomplete](https://mlsanity.com/paper/2505.23506/) (2025)
- [Learning Credal Ensembles via Distributionally Robust Optimization](https://mlsanity.com/paper/2602.08470/) (2026)
- [A Review of Bayesian Uncertainty Quantification in Deep Probabilistic Image Segmentation](https://mlsanity.com/paper/2411.16370/) (2024)
- [Reducing Aleatoric and Epistemic Uncertainty through Multi-modal Data Acquisition](https://mlsanity.com/paper/2501.18268/) (2025)
- [Evidential Uncertainty Estimation for Multi-Modal Trajectory Prediction](https://mlsanity.com/paper/2503.05274/) (2025)
- [laplax -- Laplace Approximations with JAX](https://mlsanity.com/paper/2507.17013/) (2025)
- [Two Axes of LLM Abstention: Answer Correctness and Question Answerability](https://mlsanity.com/paper/2607.08456/) (2026)
- [Measuring Orthogonality as the Blind-Spot of Uncertainty Disentanglement](https://mlsanity.com/paper/2408.12175/) (2024)
- [Benchmarking Uncertainty and its Disentanglement in multi-label Chest X-Ray Classification](https://mlsanity.com/paper/2508.04457/) (2025)
- [Uncertainty-Aware Deep Learning for Wildfire Danger Forecasting](https://mlsanity.com/paper/2509.25017/) (2025)
- [Decoupling of neural network calibration measures](https://mlsanity.com/paper/2406.02411/) (2024)
- [Fully Heteroscedastic Count Regression with Deep Double Poisson Networks](https://mlsanity.com/paper/2406.09262/) (2024)
- [How to Leverage Predictive Uncertainty Estimates for Reducing Catastrophic Forgetting in Online Continual Learning](https://mlsanity.com/paper/2407.07668/) (2024)
- [Enhancing Out-of-Distribution Detection with Extended Logit Normalization](https://mlsanity.com/paper/2504.11434/) (2025)
- [Uncertainty quantification with approximate variational learning for wearable photoplethysmography prediction tasks](https://mlsanity.com/paper/2505.11412/) (2025)
- [Generative AI for Autonomous Driving: A Review](https://mlsanity.com/paper/2505.15863/) (2025)
- [Asymmetric Duos: Sidekicks Improve Uncertainty](https://mlsanity.com/paper/2505.18636/) (2025)

## Abstract (arXiv)

Uncertainty quantification, once a singular task, has evolved into a spectrum of tasks, including abstained prediction, out-of-distribution detection, and aleatoric uncertainty quantification. The latest goal is disentanglement: the construction of multiple estimators that are each tailored to one and only one source of uncertainty. This paper presents the first benchmark of uncertainty disentanglement. We reimplement and evaluate a comprehensive range of uncertainty estimators, from Bayesian over evidential to deterministic ones, across a diverse range of uncertainty tasks on ImageNet. We find that, despite recent theoretical endeavors, no existing approach provides pairs of disentangled uncertainty estimators in practice. We further find that specialized uncertainty tasks are harder than predictive uncertainty tasks, where we observe saturating performance. Our results provide both practical advice for which uncertainty estimators to use for which specific task, and reveal opportunities for future research toward task-centric and disentangled uncertainties. All our reimplementations and Weights & Biases logs are available at https://github.com/bmucsanyi/untangle.