mlsanityData to 7 Oct 2026

ValUES: A Framework for Systematic Validation of Uncertainty Estimation in Semantic Segmentation

arXiv 2401.08501 ↗First posted 16 Jan 2024cs.CVcited by 25

Not on the map. Why?

Abstract (arXiv)

Uncertainty estimation is an essential and heavily-studied component for the reliable application of semantic segmentation methods. While various studies exist claiming methodological advances on the one hand, and successful application on the other hand, the field is currently hampered by a gap between theory and practice leaving fundamental questions unanswered: Can data-related and model-related uncertainty really be separated in practice? Which components of an uncertainty method are essential for real-world performance? Which uncertainty method works well for which application? In this work, we link this research gap to a lack of systematic and comprehensive evaluation of uncertainty methods. Specifically, we identify three key pitfalls in current literature and present an evaluation framework that bridges the research gap by providing 1) a controlled environment for studying data ambiguities as well as distribution shifts, 2) systematic ablations of relevant method components, and 3) test-beds for the five predominant uncertainty applications: OoD-detection, active learning, failure detection, calibration, and ambiguity modeling. Empirical results on simulated as well as real-world data demonstrate how the proposed framework is able to answer the predominant questions in the field revealing for instance that 1) separation of uncertainty types works on simulated data but does not necessarily translate to real-world data, 2) aggregation of scores is a crucial but currently neglected component of uncertainty methods, 3) While ensembles are performing most robustly across the different downstream tasks and settings, test-time augmentation often constitutes a light-weight alternative. Code is at: https://github.com/IML-DKFZ/values

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 32 of its 46 references that are in the map. First posted on arXiv 16 Jan 2024.

Builds on

32 of 46 references.

References · 32

  1. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  2. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  3. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
  4. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  5. 2019Deep High-Resolution Representation Learning for Visual Recognitioncited by 1,417
  6. 2016Playing for Data: Ground Truth from Computer Gamescited by 816
  7. 2018A Probabilistic U-Net for Segmentation of Ambiguous Imagescited by 281
  8. 2018Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networkscited by 278
  9. 2021Deep Deterministic Uncertainty: A Simple Baselinecited by 200
  10. 2019Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Visioncited by 182
  11. 2019Confidence Calibration and Predictive Uncertainty Estimation for Deep Medical Image Segmentationcited by 162
  12. 2018Evaluating Bayesian Deep Learning Methods for Semantic Segmentationcited by 143
20 more
  1. 2019PHiSeg: Capturing Uncertainty in Medical Image Segmentationcited by 128
  2. 2018Bias-Reduced Uncertainty Estimation for Deep Neural Classifierscited by 106
  3. 2020Stochastic Segmentation Networks: Modelling Spatially Correlated Aleatoric Uncertaintycited by 100
  4. 2019DropConnect Is Effective in Modeling Uncertainty of Bayesian Deep Networkscited by 92
  5. 2019Sampling-free Epistemic Uncertainty Estimation Using Approximated Variance Propagationcited by 74
  6. 2021On the Practicality of Deterministic Epistemic Uncertaintycited by 62
  7. 2022A Call to Reflect on Evaluation Practices for Failure Detection in Image Classificationcited by 51
  8. 2018CEREALS - Cost-Effective REgion-based Active Learning for Semantic Segmentationcited by 47
  9. 2019Supervised Uncertainty Quantification for Segmentation with Multiple Annotationscited by 42
  10. 2019Parting with Illusions about Deep Active Learningcited by 38
  11. 2021Is segmentation uncertainty useful?cited by 37
  12. 2020DEAL: Difficulty-aware Active Learning for Semantic Segmentationcited by 32
  13. 2023Navigating the Pitfalls of Active Learning Evaluation: A Systematic Framework for Meaningful Performance Assessmentcited by 25
  14. 2021Deep Deterministic Uncertainty for Semantic Segmentationcited by 24
  15. 2020MetaBox+: A new Region Based Active Learning Method for Semantic Segmentation using Priority Mapscited by 23
  16. 2021Detecting when pre-trained nnU-Net models fail silently for Covid-19 lung lesion segmentationcited by 19
  17. 2020Uncertainty Evaluation Metric for Brain Tumour Segmentationcited by 11
  18. 2023Best Practices in Active Learning for Semantic Segmentationcited by 10
  19. 2022Improving Uncertainty-based Out-of-Distribution Detection for Medical Image Segmentationcited by 5
  20. 2022Uncertainty categories in medical image segmentation: a study of source-related diversitycited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 4; 2026: 9.

25 papers.

Sort:

  1. 2024Uncertainty in Graph Neural Networks: A Surveycited by 20
  2. 2024Comparative Benchmarking of Failure Detection Methods in Medical Image Segmentation: Unveiling the Role of Confidence Aggregationcited by 12
  3. 2025Uncertainty Quantification for Machine Learning in Healthcare: A Surveycited by 11
  4. 2024Conformal Performance Range Prediction for Segmentation Output Quality Controlcited by 8
  5. 2024A Review of Bayesian Uncertainty Quantification in Deep Probabilistic Image Segmentationcited by 6
  6. 2024Arctique: An artificial histopathological dataset unifying realism and controllability for uncertainty quantificationcited by 4
  7. 2025Benchmarking Uncertainty and its Disentanglement in multi-label Chest X-Ray Classificationcited by 4
  8. 2024Uni-SLAM: Uncertainty-Aware Neural Implicit SLAM for Real-Time Dense Indoor Scene Reconstructioncited by 3
  9. 2024EDUE: Expert Disagreement-Guided One-Pass Uncertainty Estimation for Medical Image Segmentationcited by 2
  10. 2025Uncertainty-Aware Segmentation Quality Prediction via Deep Learning Bayesian Modeling: Comprehensive Evaluation and Interpretation on Skin Cancer and Liver Segmentationcited by 2
  11. 2024Soft Dice Confidence: A Near-Optimal Confidence Estimator for Selective Prediction in Semantic Segmentationcited by 1
  12. 2024REPEAT: Improving Uncertainty Estimation in Representation Learning Explainabilitycited by 1
13 more
  1. 2025Uncertainty evaluation of segmentation models for Earth observationcited by 1
  2. 2026Rethinking Uncertainty Quantification and Entanglement in Image Segmentationcited by 1
  3. 2026Better than Average: Spatially-Aware Aggregation of Segmentation Uncertainty Improves Downstream Performancecited by 1
  4. 2026Lost in the Folds: When Cross-Validation Is Not a Deep Ensemble for Uncertainty Estimationcited by 1
  5. 2024Probabilistic Contrastive Learning with Explicit Concentration on the Hyperspherecited by 0
  6. 2024Improving EO Foundation Models with Confidence Assessment for enhanced Semantic segmentationcited by 0
  7. 2024Instance-wise Uncertainty for Class Imbalance in Semantic Segmentationcited by 0
  8. 2026Budget-Aware Uncertainty for Radiotherapy Segmentation QA Using nnU-Netcited by 0
  9. 2026U-SEG: Uncertainty in SEGmentation -- A systematic multi-variable explorationcited by 0
  10. 2026Rethinking Post-Hoc Calibration in Semantic Segmentationcited by 0
  11. 2026GRC-ProbNet: Uncertainty-aware Feature Extraction for Cardiovascular Disease Classificationcited by 0
  12. 2026Towards Trustworthy AI for Glioma Diagnosis: A Task-Aware Evaluation of Uncertainty Quantificationcited by 0
  13. 2026Structural Limits of the Information-Theoretic Uncertainty Decompositioncited by 0