mlsanityData to 7 Oct 2026

Comparative Benchmarking of Failure Detection Methods in Medical Image Segmentation: Unveiling the Role of Confidence Aggregation

arXiv 2406.03323 ↗First posted 5 Jun 2024cs.CVcited by 12

Not on the map. Why?

Abstract (arXiv)

Semantic segmentation is an essential component of medical image analysis research, with recent deep learning algorithms offering out-of-the-box applicability across diverse datasets. Despite these advancements, segmentation failures remain a significant concern for real-world clinical applications, necessitating reliable detection mechanisms. This paper introduces a comprehensive benchmarking framework aimed at evaluating failure detection methodologies within medical image segmentation. Through our analysis, we identify the strengths and limitations of current failure detection metrics, advocating for the risk-coverage analysis as a holistic evaluation approach. Utilizing a collective dataset comprising five public 3D medical image collections, we assess the efficacy of various failure detection strategies under realistic test-time distribution shifts. Our findings highlight the importance of pixel confidence aggregation and we observe superior performance of the pairwise Dice score (Roy et al., 2019) between ensemble predictions, positioning it as a simple and robust baseline for failure detection in medical image segmentation. To promote ongoing research, we make the benchmarking framework available to the community.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 50 of its 82 references that are in the map. First posted on arXiv 5 Jun 2024.

Builds on

50 of 82 references.

References · 50

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  5. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  6. 2017Selective Classification for Deep Neural Networkscited by 799
  7. 2018Evidential Deep Learning to Quantify Classification Uncertaintycited by 728
  8. 2021The Medical Segmentation Decathloncited by 620
  9. 2018Predictive Uncertainty Estimation via Prior Networkscited by 555
  10. 2018Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challengecited by 508
  11. 2022MONAI: An open-source framework for deep learning in healthcarecited by 438
  12. 2018Confounding variables can degrade generalization performance of radiological deep learning modelscited by 374
38 more
  1. 2019A large annotated medical image dataset for the development and evaluation of segmentation algorithmscited by 371
  2. 2018A Probabilistic U-Net for Segmentation of Ambiguous Imagescited by 281
  3. 2018Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networkscited by 278
  4. 2015Bayesian SegNet: Model Uncertainty in Deep Convolutional Encoder-Decoder Architectures for Scene Understandingcited by 218
  5. 2022Metrics reloaded: Recommendations for image analysis validationcited by 190
  6. 2019The state of the art in kidney and kidney tumor segmentation in contrast-enhanced CT imaging: Results of the KiTS19 Challengecited by 188
  7. 2020TorchIO: A Python library for efficient loading, preprocessing, augmentation and patch-based sampling of medical images in deep learningcited by 176
  8. 2020Efficient and Scalable Bayesian Neural Nets with Rank-1 Factorscited by 171
  9. 2019Confidence Calibration and Predictive Uncertainty Estimation for Deep Medical Image Segmentationcited by 162
  10. 2021A Unified Survey on Anomaly, Novelty, Open-Set, and Out-of-Distribution Detection: Solutions and Future Challengescited by 154
  11. 2018Exploring Uncertainty Measures in Deep Networks for Multiple Sclerosis Lesion Detection and Segmentationcited by 146
  12. 2020Shape-aware Meta-learning for Generalizing Prostate MRI Segmentation to Unseen Domainscited by 124
  13. 2020MosMedData: Chest CT Scans With COVID-19 Related Findings Datasetcited by 103
  14. 2023The KiTS21 Challenge: Automatic segmentation of kidneys, renal tumors, and renal cysts in corticomedullary-phase CTcited by 101
  15. 2020Stochastic Segmentation Networks: Modelling Spatially Correlated Aleatoric Uncertaintycited by 100
  16. 2020Synthesize then Compare: Detecting Failures and Anomalies for Semantic Segmentationcited by 96
  17. 2018Deep learning to achieve clinically applicable segmentation of head and neck anatomy for radiotherapycited by 76
  18. 2022Trustworthy clinical AI solutions: a unified review of uncertainty quantification in deep learning models for medical image analysiscited by 62
  19. 2022A Call to Reflect on Evaluation Practices for Failure Detection in Image Classificationcited by 51
  20. 2018Deep Learning Predicts Hip Fracture using Confounding Patient and Healthcare Variablescited by 50
  21. 2021The Federated Tumor Segmentation (FeTS) Challengecited by 49
  22. 2018Leveraging Uncertainty Estimates for Predicting Segmentation Qualitycited by 44
  23. 2018Bayesian QuickNAT: Model Uncertainty in Deep Whole-Brain Segmentation for Structure-wise Quality Controlcited by 39
  24. 2017Reverse Classification Accuracy: Predicting Segmentation Performance in the Absence of Ground Truthcited by 35
  25. 2023What Can We Learn From The Selective Prediction And Uncertainty Estimation Performance Of 523 Imagenet Classifierscited by 35
  26. 2022TBraTS: Trusted Brain Tumor Segmentationcited by 30
  27. 2020Studying Robustness of Semantic Segmentation under Domain Shift in cardiac MRIcited by 28
  28. 2024ValUES: A Framework for Systematic Validation of Uncertainty Estimation in Semantic Segmentationcited by 25
  29. 2018Real-time Prediction of Segmentation Qualitycited by 23
  30. 2020Deep Generative Model-based Quality Control for Cardiac MRI Segmentationcited by 21
  31. 2020Estimating Uncertainty in Neural Networks for Cardiac MRI Segmentation: A Benchmark Studycited by 19
  32. 2022Efficient Bayesian Uncertainty Estimation for nnU-Netcited by 19
  33. 2022Transformer-based out-of-distribution detection for clinically safe segmentationcited by 16
  34. 2022Layer Ensembles: A Single-Pass Uncertainty Estimation in Deep Learning for Segmentationcited by 15
  35. 2020Uncertainty Evaluation Metric for Brain Tumour Segmentationcited by 11
  36. 2022Improved post-hoc probability calibration for out-of-domain MRI segmentationcited by 5
  37. 2023Benchmarking Scalable Epistemic Uncertainty Quantification in Organ Segmentationcited by 4
  38. 2023Redesigning Out-of-Distribution Detection on 3D Medical Imagescited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 7.

12 papers.

  1. 2025A Large Scale Benchmark for Test Time Adaptation Methods in Medical Image Segmentationcited by 5
  2. 2025Advances in Automated Fetal Brain MRI Segmentation and Biometry: Insights from the FeTA 2024 Challengecited by 4
  3. 2026Better than Average: Spatially-Aware Aggregation of Segmentation Uncertainty Improves Downstream Performancecited by 1
  4. 2026Lost in the Folds: When Cross-Validation Is Not a Deep Ensemble for Uncertainty Estimationcited by 1
  5. 2025Unsupervised Source-Free Ranking of Biomedical Segmentation Models Under Distribution Shiftcited by 0
  6. 2025Reliable Radiologic Skeletal Muscle Area Assessment -- A Biomarker for Cancer Cachexia Diagnosiscited by 0
  7. 2025Can General-Purpose Omnimodels Compete with Specialists? A Case Study in Medical Image Segmentationcited by 0
  8. 2026Region-Normalized DPO for Medical Image Segmentation under Noisy Judgescited by 0
  9. 2026Rethinking Post-Hoc Calibration in Semantic Segmentationcited by 0
  10. 2026Self-Auditing Residual Drifting for Pathology-Preserving Accelerated Knee MRIcited by 0
  11. 2026From Few-Shot Segmentation to Clinician-in-the-Loop Medical Image Analysiscited by 0
  12. 2026Towards Trustworthy AI for Glioma Diagnosis: A Task-Aware Evaluation of Uncertainty Quantificationcited by 0