# Confidence intervals uncovered: Are we ready for real-world medical imaging AI?

arXiv: https://arxiv.org/abs/2409.17763 · HTML: https://mlsanity.com/paper/2409.17763/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 19 references that are in the map.
- First posted on arXiv 26 Sep 2024.

## Builds on (most cited first, 7 of 7)

- [The Medical Segmentation Decathlon](https://mlsanity.com/paper/2106.05735/) (2021)
- [A large annotated medical image dataset for the development and evaluation of segmentation algorithms](https://mlsanity.com/paper/1902.09063/) (2019)
- [Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)](https://mlsanity.com/paper/2003.12206/) (2020)
- [Accounting for Variance in Machine Learning Benchmarks](https://mlsanity.com/paper/2103.03098/) (2021)
- [Time for a change: a tutorial for comparing multiple classifiers through Bayesian analysis](https://mlsanity.com/paper/1606.04316/) (2016)
- [Why rankings of biomedical image analysis competitions should be interpreted with care](https://mlsanity.com/paper/1806.02051/) (2018)
- [Methods and open-source toolkit for analyzing and visualizing challenge results](https://mlsanity.com/paper/1910.05121/) (2019)

## Built on by (most cited first, 12 of 12)

- [False Promises in Medical Imaging AI? Assessing Validity of Outperformance Claims](https://mlsanity.com/paper/2505.04720/) (2025)
- [Subgroup Performance Analysis in Hidden Stratifications](https://mlsanity.com/paper/2503.10382/) (2025)
- [Auditing Significance, Metric Choice, and Demographic Fairness in Medical AI Challenges](https://mlsanity.com/paper/2512.19091/) (2025)
- [Implicit Deformable Medical Image Registration with Learnable Kernels](https://mlsanity.com/paper/2506.02150/) (2025)
- [OxEnsemble: Fair Ensembles for Low-Data Classification](https://mlsanity.com/paper/2512.09665/) (2025)
- [GP-VM×SMA: Benchmarking General-Purpose Vision Models and Specialized Architectures for 2D Medical Image Segmentation](https://mlsanity.com/paper/2603.13044/) (2026)
- [Self-Supervised Multi-Stage Domain Unlearning for White-Matter Lesion Segmentation](https://mlsanity.com/paper/2603.13328/) (2026)
- [Fairness Beyond Demographics: Optimizing Performance Across Appearance-Based Hidden Cohorts in Medical Imaging](https://mlsanity.com/paper/2605.29827/) (2026)
- [Crossing the Validation Crisis: Cross-Validation Reduces Benchmarking Variance Surprisingly Well](https://mlsanity.com/paper/2606.12552/) (2026)
- [Towards Global AI-Driven Cervical Cancer Screening](https://mlsanity.com/paper/2606.15019/) (2026)
- [MedSegBenchmarker: A Raw-Count-First Framework for Controlled 2D Medical Image Segmentation Benchmarks](https://mlsanity.com/paper/2608.29677/) (2026)
- [MIRTO: a registration-gated, multiverse-tested evaluation protocol for unsupervised anomaly segmentation in brain MRI](https://mlsanity.com/paper/2610.02136/) (2026)

## Abstract (arXiv)

Medical imaging is spearheading the AI transformation of healthcare. Performance reporting is key to determine which methods should be translated into clinical practice. Frequently, broad conclusions are simply derived from mean performance values. In this paper, we argue that this common practice is often a misleading simplification as it ignores performance variability. Our contribution is threefold. (1) Analyzing all MICCAI segmentation papers (n = 221) published in 2023, we first observe that more than 50% of papers do not assess performance variability at all. Moreover, only one (0.5%) paper reported confidence intervals (CIs) for model performance. (2) To address the reporting bottleneck, we show that the unreported standard deviation (SD) in segmentation papers can be approximated by a second-order polynomial function of the mean Dice similarity coefficient (DSC). Based on external validation data from 56 previous MICCAI challenges, we demonstrate that this approximation can accurately reconstruct the CI of a method using information provided in publications. (3) Finally, we reconstructed 95% CIs around the mean DSC of MICCAI 2023 segmentation papers. The median CI width was 0.03 which is three times larger than the median performance gap between the first and second ranked method. For more than 60% of papers, the mean performance of the second-ranked method was within the CI of the first-ranked method. We conclude that current publications typically do not provide sufficient evidence to support which models could potentially be translated into clinical practice.