# ValUES: A Framework for Systematic Validation of Uncertainty Estimation in Semantic Segmentation

arXiv: https://arxiv.org/abs/2401.08501 · HTML: https://mlsanity.com/paper/2401.08501/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 46 references that are in the map.
- First posted on arXiv 16 Jan 2024.

## Builds on (most cited first, 30 of 32)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [Playing for Data: Ground Truth from Computer Games](https://mlsanity.com/paper/1608.02192/) (2016)
- [A Probabilistic U-Net for Segmentation of Ambiguous Images](https://mlsanity.com/paper/1806.05034/) (2018)
- [Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks](https://mlsanity.com/paper/1807.07356/) (2018)
- [Deep Deterministic Uncertainty: A Simple Baseline](https://mlsanity.com/paper/2102.11582/) (2021)
- [Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Vision](https://mlsanity.com/paper/1906.01620/) (2019)
- [Confidence Calibration and Predictive Uncertainty Estimation for Deep Medical Image Segmentation](https://mlsanity.com/paper/1911.13273/) (2019)
- [Evaluating Bayesian Deep Learning Methods for Semantic Segmentation](https://mlsanity.com/paper/1811.12709/) (2018)
- [PHiSeg: Capturing Uncertainty in Medical Image Segmentation](https://mlsanity.com/paper/1906.04045/) (2019)
- [Bias-Reduced Uncertainty Estimation for Deep Neural Classifiers](https://mlsanity.com/paper/1805.08206/) (2018)
- [Stochastic Segmentation Networks: Modelling Spatially Correlated Aleatoric Uncertainty](https://mlsanity.com/paper/2006.06015/) (2020)
- [DropConnect Is Effective in Modeling Uncertainty of Bayesian Deep Networks](https://mlsanity.com/paper/1906.04569/) (2019)
- [Sampling-free Epistemic Uncertainty Estimation Using Approximated Variance Propagation](https://mlsanity.com/paper/1908.00598/) (2019)
- [On the Practicality of Deterministic Epistemic Uncertainty](https://mlsanity.com/paper/2107.00649/) (2021)
- [A Call to Reflect on Evaluation Practices for Failure Detection in Image Classification](https://mlsanity.com/paper/2211.15259/) (2022)
- [CEREALS - Cost-Effective REgion-based Active Learning for Semantic Segmentation](https://mlsanity.com/paper/1810.09726/) (2018)
- [Supervised Uncertainty Quantification for Segmentation with Multiple Annotations](https://mlsanity.com/paper/1907.01949/) (2019)
- [Parting with Illusions about Deep Active Learning](https://mlsanity.com/paper/1912.05361/) (2019)
- [Is segmentation uncertainty useful?](https://mlsanity.com/paper/2103.16265/) (2021)
- [DEAL: Difficulty-aware Active Learning for Semantic Segmentation](https://mlsanity.com/paper/2010.08705/) (2020)
- [Navigating the Pitfalls of Active Learning Evaluation: A Systematic Framework for Meaningful Performance Assessment](https://mlsanity.com/paper/2301.10625/) (2023)
- [Deep Deterministic Uncertainty for Semantic Segmentation](https://mlsanity.com/paper/2111.00079/) (2021)
- [MetaBox+: A new Region Based Active Learning Method for Semantic Segmentation using Priority Maps](https://mlsanity.com/paper/2010.01884/) (2020)
- [Detecting when pre-trained nnU-Net models fail silently for Covid-19 lung lesion segmentation](https://mlsanity.com/paper/2107.05975/) (2021)
- [Uncertainty Evaluation Metric for Brain Tumour Segmentation](https://mlsanity.com/paper/2005.14262/) (2020)
- [Best Practices in Active Learning for Semantic Segmentation](https://mlsanity.com/paper/2302.04075/) (2023)

## Built on by (most cited first, 25 of 25)

- [Uncertainty in Graph Neural Networks: A Survey](https://mlsanity.com/paper/2403.07185/) (2024)
- [Uncertainty Quantification for Machine Learning in Healthcare: A Survey](https://mlsanity.com/paper/2505.02874/) (2025)
- [Comparative Benchmarking of Failure Detection Methods in Medical Image Segmentation: Unveiling the Role of Confidence Aggregation](https://mlsanity.com/paper/2406.03323/) (2024)
- [Conformal Performance Range Prediction for Segmentation Output Quality Control](https://mlsanity.com/paper/2407.13307/) (2024)
- [A Review of Bayesian Uncertainty Quantification in Deep Probabilistic Image Segmentation](https://mlsanity.com/paper/2411.16370/) (2024)
- [Arctique: An artificial histopathological dataset unifying realism and controllability for uncertainty quantification](https://mlsanity.com/paper/2411.07097/) (2024)
- [Benchmarking Uncertainty and its Disentanglement in multi-label Chest X-Ray Classification](https://mlsanity.com/paper/2508.04457/) (2025)
- [Uni-SLAM: Uncertainty-Aware Neural Implicit SLAM for Real-Time Dense Indoor Scene Reconstruction](https://mlsanity.com/paper/2412.00242/) (2024)
- [EDUE: Expert Disagreement-Guided One-Pass Uncertainty Estimation for Medical Image Segmentation](https://mlsanity.com/paper/2403.16594/) (2024)
- [Uncertainty-Aware Segmentation Quality Prediction via Deep Learning Bayesian Modeling: Comprehensive Evaluation and Interpretation on Skin Cancer and Liver Segmentation](https://mlsanity.com/paper/2508.01460/) (2025)
- [Soft Dice Confidence: A Near-Optimal Confidence Estimator for Selective Prediction in Semantic Segmentation](https://mlsanity.com/paper/2402.10665/) (2024)
- [REPEAT: Improving Uncertainty Estimation in Representation Learning Explainability](https://mlsanity.com/paper/2412.08513/) (2024)
- [Uncertainty evaluation of segmentation models for Earth observation](https://mlsanity.com/paper/2510.19586/) (2025)
- [Rethinking Uncertainty Quantification and Entanglement in Image Segmentation](https://mlsanity.com/paper/2603.18792/) (2026)
- [Better than Average: Spatially-Aware Aggregation of Segmentation Uncertainty Improves Downstream Performance](https://mlsanity.com/paper/2603.29941/) (2026)
- [Lost in the Folds: When Cross-Validation Is Not a Deep Ensemble for Uncertainty Estimation](https://mlsanity.com/paper/2605.18329/) (2026)
- [Probabilistic Contrastive Learning with Explicit Concentration on the Hypersphere](https://mlsanity.com/paper/2405.16460/) (2024)
- [Improving EO Foundation Models with Confidence Assessment for enhanced Semantic segmentation](https://mlsanity.com/paper/2406.18279/) (2024)
- [Instance-wise Uncertainty for Class Imbalance in Semantic Segmentation](https://mlsanity.com/paper/2407.12609/) (2024)
- [Budget-Aware Uncertainty for Radiotherapy Segmentation QA Using nnU-Net](https://mlsanity.com/paper/2604.11798/) (2026)
- [U-SEG: Uncertainty in SEGmentation -- A systematic multi-variable exploration](https://mlsanity.com/paper/2605.15421/) (2026)
- [Rethinking Post-Hoc Calibration in Semantic Segmentation](https://mlsanity.com/paper/2607.01902/) (2026)
- [GRC-ProbNet: Uncertainty-aware Feature Extraction for Cardiovascular Disease Classification](https://mlsanity.com/paper/2607.10357/) (2026)
- [Towards Trustworthy AI for Glioma Diagnosis: A Task-Aware Evaluation of Uncertainty Quantification](https://mlsanity.com/paper/2609.39429/) (2026)
- [Structural Limits of the Information-Theoretic Uncertainty Decomposition](https://mlsanity.com/paper/2609.39591/) (2026)

## Abstract (arXiv)

Uncertainty estimation is an essential and heavily-studied component for the reliable application of semantic segmentation methods. While various studies exist claiming methodological advances on the one hand, and successful application on the other hand, the field is currently hampered by a gap between theory and practice leaving fundamental questions unanswered: Can data-related and model-related uncertainty really be separated in practice? Which components of an uncertainty method are essential for real-world performance? Which uncertainty method works well for which application? In this work, we link this research gap to a lack of systematic and comprehensive evaluation of uncertainty methods. Specifically, we identify three key pitfalls in current literature and present an evaluation framework that bridges the research gap by providing 1) a controlled environment for studying data ambiguities as well as distribution shifts, 2) systematic ablations of relevant method components, and 3) test-beds for the five predominant uncertainty applications: OoD-detection, active learning, failure detection, calibration, and ambiguity modeling. Empirical results on simulated as well as real-world data demonstrate how the proposed framework is able to answer the predominant questions in the field revealing for instance that 1) separation of uncertainty types works on simulated data but does not necessarily translate to real-world data, 2) aggregation of scores is a crucial but currently neglected component of uncertainty methods, 3) While ensembles are performing most robustly across the different downstream tasks and settings, test-time augmentation often constitutes a light-weight alternative. Code is at: https://github.com/IML-DKFZ/values