# Comparative Benchmarking of Failure Detection Methods in Medical Image Segmentation: Unveiling the Role of Confidence Aggregation

arXiv: https://arxiv.org/abs/2406.03323 · HTML: https://mlsanity.com/paper/2406.03323/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 82 references that are in the map.
- First posted on arXiv 5 Jun 2024.

## Builds on (most cited first, 30 of 50)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Selective Classification for Deep Neural Networks](https://mlsanity.com/paper/1705.08500/) (2017)
- [Evidential Deep Learning to Quantify Classification Uncertainty](https://mlsanity.com/paper/1806.01768/) (2018)
- [The Medical Segmentation Decathlon](https://mlsanity.com/paper/2106.05735/) (2021)
- [Predictive Uncertainty Estimation via Prior Networks](https://mlsanity.com/paper/1802.10501/) (2018)
- [Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challenge](https://mlsanity.com/paper/1811.02629/) (2018)
- [MONAI: An open-source framework for deep learning in healthcare](https://mlsanity.com/paper/2211.02701/) (2022)
- [Confounding variables can degrade generalization performance of radiological deep learning models](https://mlsanity.com/paper/1807.00431/) (2018)
- [A large annotated medical image dataset for the development and evaluation of segmentation algorithms](https://mlsanity.com/paper/1902.09063/) (2019)
- [A Probabilistic U-Net for Segmentation of Ambiguous Images](https://mlsanity.com/paper/1806.05034/) (2018)
- [Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks](https://mlsanity.com/paper/1807.07356/) (2018)
- [Bayesian SegNet: Model Uncertainty in Deep Convolutional Encoder-Decoder Architectures for Scene Understanding](https://mlsanity.com/paper/1511.02680/) (2015)
- [Metrics reloaded: Recommendations for image analysis validation](https://mlsanity.com/paper/2206.01653/) (2022)
- [The state of the art in kidney and kidney tumor segmentation in contrast-enhanced CT imaging: Results of the KiTS19 Challenge](https://mlsanity.com/paper/1912.01054/) (2019)
- [TorchIO: A Python library for efficient loading, preprocessing, augmentation and patch-based sampling of medical images in deep learning](https://mlsanity.com/paper/2003.04696/) (2020)
- [Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors](https://mlsanity.com/paper/2005.07186/) (2020)
- [Confidence Calibration and Predictive Uncertainty Estimation for Deep Medical Image Segmentation](https://mlsanity.com/paper/1911.13273/) (2019)
- [A Unified Survey on Anomaly, Novelty, Open-Set, and Out-of-Distribution Detection: Solutions and Future Challenges](https://mlsanity.com/paper/2110.14051/) (2021)
- [Exploring Uncertainty Measures in Deep Networks for Multiple Sclerosis Lesion Detection and Segmentation](https://mlsanity.com/paper/1808.01200/) (2018)
- [Shape-aware Meta-learning for Generalizing Prostate MRI Segmentation to Unseen Domains](https://mlsanity.com/paper/2007.02035/) (2020)
- [MosMedData: Chest CT Scans With COVID-19 Related Findings Dataset](https://mlsanity.com/paper/2005.06465/) (2020)
- [The KiTS21 Challenge: Automatic segmentation of kidneys, renal tumors, and renal cysts in corticomedullary-phase CT](https://mlsanity.com/paper/2307.01984/) (2023)
- [Stochastic Segmentation Networks: Modelling Spatially Correlated Aleatoric Uncertainty](https://mlsanity.com/paper/2006.06015/) (2020)
- [Synthesize then Compare: Detecting Failures and Anomalies for Semantic Segmentation](https://mlsanity.com/paper/2003.08440/) (2020)
- [Deep learning to achieve clinically applicable segmentation of head and neck anatomy for radiotherapy](https://mlsanity.com/paper/1809.04430/) (2018)
- [Trustworthy clinical AI solutions: a unified review of uncertainty quantification in deep learning models for medical image analysis](https://mlsanity.com/paper/2210.03736/) (2022)

## Built on by (most cited first, 12 of 12)

- [Advances in Automated Fetal Brain MRI Segmentation and Biometry: Insights from the FeTA 2024 Challenge](https://mlsanity.com/paper/2505.02784/) (2025)
- [A Large Scale Benchmark for Test Time Adaptation Methods in Medical Image Segmentation](https://mlsanity.com/paper/2512.02497/) (2025)
- [Better than Average: Spatially-Aware Aggregation of Segmentation Uncertainty Improves Downstream Performance](https://mlsanity.com/paper/2603.29941/) (2026)
- [Lost in the Folds: When Cross-Validation Is Not a Deep Ensemble for Uncertainty Estimation](https://mlsanity.com/paper/2605.18329/) (2026)
- [Unsupervised Source-Free Ranking of Biomedical Segmentation Models Under Distribution Shift](https://mlsanity.com/paper/2503.00450/) (2025)
- [Reliable Radiologic Skeletal Muscle Area Assessment -- A Biomarker for Cancer Cachexia Diagnosis](https://mlsanity.com/paper/2503.16556/) (2025)
- [Can General-Purpose Omnimodels Compete with Specialists? A Case Study in Medical Image Segmentation](https://mlsanity.com/paper/2509.00866/) (2025)
- [Region-Normalized DPO for Medical Image Segmentation under Noisy Judges](https://mlsanity.com/paper/2601.23222/) (2026)
- [Rethinking Post-Hoc Calibration in Semantic Segmentation](https://mlsanity.com/paper/2607.01902/) (2026)
- [Self-Auditing Residual Drifting for Pathology-Preserving Accelerated Knee MRI](https://mlsanity.com/paper/2607.02428/) (2026)
- [From Few-Shot Segmentation to Clinician-in-the-Loop Medical Image Analysis](https://mlsanity.com/paper/2609.10001/) (2026)
- [Towards Trustworthy AI for Glioma Diagnosis: A Task-Aware Evaluation of Uncertainty Quantification](https://mlsanity.com/paper/2609.39429/) (2026)

## Abstract (arXiv)

Semantic segmentation is an essential component of medical image analysis research, with recent deep learning algorithms offering out-of-the-box applicability across diverse datasets. Despite these advancements, segmentation failures remain a significant concern for real-world clinical applications, necessitating reliable detection mechanisms. This paper introduces a comprehensive benchmarking framework aimed at evaluating failure detection methodologies within medical image segmentation. Through our analysis, we identify the strengths and limitations of current failure detection metrics, advocating for the risk-coverage analysis as a holistic evaluation approach. Utilizing a collective dataset comprising five public 3D medical image collections, we assess the efficacy of various failure detection strategies under realistic test-time distribution shifts. Our findings highlight the importance of pixel confidence aggregation and we observe superior performance of the pairwise Dice score (Roy et al., 2019) between ensemble predictions, positioning it as a simple and robust baseline for failure detection in medical image segmentation. To promote ongoing research, we make the benchmarking framework available to the community.