mlsanityData to 7 Oct 2026

Benchmarking Uncertainty Disentanglement: Specialized Uncertainties for Specialized Tasks

arXiv 2402.19460 ↗First posted 29 Feb 2024cs.LG stat.MLcited by 78

Not on the map. Why?

Abstract (arXiv)

Uncertainty quantification, once a singular task, has evolved into a spectrum of tasks, including abstained prediction, out-of-distribution detection, and aleatoric uncertainty quantification. The latest goal is disentanglement: the construction of multiple estimators that are each tailored to one and only one source of uncertainty. This paper presents the first benchmark of uncertainty disentanglement. We reimplement and evaluate a comprehensive range of uncertainty estimators, from Bayesian over evidential to deterministic ones, across a diverse range of uncertainty tasks on ImageNet. We find that, despite recent theoretical endeavors, no existing approach provides pairs of disentangled uncertainty estimators in practice. We further find that specialized uncertainty tasks are harder than predictive uncertainty tasks, where we observe saturating performance. Our results provide both practical advice for which uncertainty estimators to use for which specific task, and reveal opportunities for future research toward task-centric and disentangled uncertainties. All our reimplementations and Weights & Biases logs are available at https://github.com/bmucsanyi/untangle.

Key facts. Cited by 78 papers in the map (data through 7 Oct 2026). Builds on 33 of its 47 references that are in the map. First posted on arXiv 29 Feb 2024.

Builds on

33 of 47 references.

References · 33

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  3. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  4. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  5. 2016Wide Residual Networkscited by 3,057
  6. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  7. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,472
  8. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  9. 2020Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awarenesscited by 389
  10. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 359
  11. 2019Learning Loss for Active Learningcited by 335
  12. 2020Are we done with ImageNet?cited by 329
21 more
  1. 2021Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotationscited by 253
  2. 2019Human uncertainty makes classification more robustcited by 231
  3. 2021Deep Deterministic Uncertainty: A Simple Baselinecited by 200
  4. 2015Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networkscited by 138
  5. 2022Quantifying Aleatoric and Epistemic Uncertainty in Machine Learning: Are Conditional Entropy and Mutual Information Appropriate Measures?cited by 124
  6. 2022Plex: Towards Reliability using Pretrained Large Model Extensionscited by 102
  7. 2022A Deeper Look into Aleatoric and Epistemic Uncertainty Disentanglementcited by 87
  8. 2021Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learningcited by 81
  9. 2021On the Practicality of Deterministic Epistemic Uncertaintycited by 62
  10. 2022Eliciting and Learning with Soft Labels from Every Annotatorcited by 43
  11. 2023On Second-Order Scoring Rules for Epistemic Uncertainty Quantificationcited by 38
  12. 2023What Can We Learn From The Selective Prediction And Uncertainty Estimation Performance Of 523 Imagenet Classifierscited by 35
  13. 2023A framework for benchmarking class-out-of-distribution detection and its application to ImageNetcited by 31
  14. 2023Probabilistic Contrastive Learning Recovers the Correct Aleatoric Uncertainty of Ambiguous Inputscited by 28
  15. 2023Sources of Uncertainty in Supervised Machine Learning -- A Statisticians' Viewcited by 27
  16. 2022Uncertainty Estimates of Predictions via a General Bias-Variance Decompositioncited by 20
  17. 2023URL: A Representation Learning Benchmark for Transferable Uncertainty Estimatescited by 17
  18. 2021Ensemble-based Uncertainty Quantification: Bayesian versus Credal Inferencecited by 16
  19. 2023Massively Scaling Heteroscedastic Classifierscited by 11
  20. 2022Ensembling over Classifiers: a Bias-Variance Perspectivecited by 6
  21. 2023Likelihood Annealing: Fast Calibrated Uncertainty for Regressioncited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 34; 2026: 31.

78 papers.

Sort:

By area:

  1. 2024Rethinking Aleatoric and Epistemic Uncertaintycited by 40
  2. 2025Position: Uncertainty Quantification Needs Reassessment for Large-language Model Agentscited by 27
  3. 2025Revisiting Uncertainty Estimation and Calibration of Large Language Modelscited by 22
  4. 2024Conformalized Credal Regions for Classification with Ambiguous Ground Truthcited by 19
  5. 2024On Information-Theoretic Measures of Predictive Uncertaintycited by 16
  6. 2025Anthropomimetic Uncertainty: What Verbalized Uncertainty in Language Models is Missingcited by 14
  7. 2024What Variables Affect Out-of-Distribution Generalization in Pretrained Models?cited by 13
  8. 2025Out-of-Distribution Segmentation in Autonomous Driving: Problems and State of the Artcited by 13
  9. 2025Query-Level Uncertainty in Large Language Modelscited by 12
  10. 2025Are We Done with Object-Centric Learning?cited by 10
  11. 2025Uncertainty Estimation by Flexible Evidential Deep Learningcited by 10
  12. 2024A Structured Review of Literature on Uncertainty in Machine Learning & Deep Learningcited by 9
38 more
  1. 2025Position: Epistemic uncertainty estimation methods are fundamentally incompletecited by 8
  2. 2025Fine-Grained Uncertainty Decomposition in Large Language Models: A Spectral Approachcited by 8
  3. 2024A Review of Bayesian Uncertainty Quantification in Deep Probabilistic Image Segmentationcited by 6
  4. 2026Learning Credal Ensembles via Distributionally Robust Optimizationcited by 6
  5. 2025Reducing Aleatoric and Epistemic Uncertainty through Multi-modal Data Acquisitioncited by 5
  6. 2025laplax -- Laplace Approximations with JAXcited by 5
  7. 2024Measuring Orthogonality as the Blind-Spot of Uncertainty Disentanglementcited by 4
  8. 2025Evidential Uncertainty Estimation for Multi-Modal Trajectory Predictioncited by 4
  9. 2025Benchmarking Uncertainty and its Disentanglement in multi-label Chest X-Ray Classificationcited by 4
  10. 2025Uncertainty-Aware Deep Learning for Wildfire Danger Forecastingcited by 4
  11. 2026Two Axes of LLM Abstention: Answer Correctness and Question Answerabilitycited by 4
  12. 2024Decoupling of neural network calibration measurescited by 3
  13. 2024Fully Heteroscedastic Count Regression with Deep Double Poisson Networkscited by 3
  14. 2024How to Leverage Predictive Uncertainty Estimates for Reducing Catastrophic Forgetting in Online Continual Learningcited by 3
  15. 2025Enhancing Out-of-Distribution Detection with Extended Logit Normalizationcited by 3
  16. 2025Uncertainty quantification with approximate variational learning for wearable photoplethysmography prediction taskscited by 3
  17. 2025Generative AI for Autonomous Driving: A Reviewcited by 3
  18. 2026Architecting Trust in Artificial Epistemic Agentscited by 3
  19. 2024Disentangling Uncertainty for Safe Social Navigation using Deep Reinforcement Learningcited by 2
  20. 2024Fixed-Mean Gaussian Processes for Post-hoc Bayesian Deep Learningcited by 2
  21. 2025Rethinking Approximate Gaussian Inference in Classificationcited by 2
  22. 2025Asymmetric Duos: Sidekicks Improve Uncertaintycited by 2
  23. 2025What Does It Take to Build a Performant Selective Classifier?cited by 2
  24. 2025Uncertainty Quantification for Regression: A Unified Framework based on kernel scorescited by 2
  25. 2026Not Just How Much, But Where: Decomposing Epistemic Uncertainty into Per-Class Contributionscited by 2
  26. 2024Uncertainties of Latent Representations in Computer Visioncited by 1
  27. 2025Trust Me, I Know the Way: Predictive Uncertainty in the Presence of Shortcut Learningcited by 1
  28. 2025Finer Disentanglement of Aleatoric Uncertainty Can Accelerate Chemical Histopathology Imagingcited by 1
  29. 2025Cooperative Variance Estimation and Bayesian Neural Networks for Disentangling Aleatoric and Epistemic Uncertaintiescited by 1
  30. 2025Uncertainty Quantification with Proper Scoring Rules: Adjusting Measures to Prediction Taskscited by 1
  31. 2025Scaling Laws for Uncertainty in Deep Learningcited by 1
  32. 2025Last Layer Hamiltonian Monte Carlocited by 1
  33. 2025Performance of Conformal Prediction in Capturing Aleatoric Uncertaintycited by 1
  34. 2025A systematic evaluation of uncertainty quantification techniques in deep learning: a case study in photoplethysmography signal analysiscited by 1
  35. 2025On the Predictive Skill of Artificial Intelligence-based Weather Models for Extreme Events using Uncertainty Quantificationcited by 1
  36. 2026Set-based v.s. Distribution-based Representations of Epistemic Uncertainty: A Comparative Studycited by 1
  37. 2026The Challenge of Out-Of-Distribution Detection in Motor Imagery BCIscited by 1
  38. 2026Rethinking Uncertainty Quantification and Entanglement in Image Segmentationcited by 1

50 most cited of 78 All 78 (JSON)