Why is the winner the best?
Not on the map. Why?
Abstract (arXiv)
International benchmarking competitions have become fundamental for the comparative performance assessment of image analysis methods. However, little attention has been given to investigating what can be learnt from these competitions. Do they really generate scientific progress? What are common and successful participation strategies? What makes a solution superior to a competing method? To address this gap in the literature, we performed a multi-center study with all 80 competitions that were conducted in the scope of IEEE ISBI 2021 and MICCAI 2021. Statistical analyses performed based on comprehensive descriptions of the submitted algorithms linked to their rank as well as the underlying participation strategies revealed common characteristics of winning solutions. These typically include the use of multi-task learning (63%) and/or multi-stage pipelines (61%), and a focus on augmentation (100%), image preprocessing (97%), data curation (79%), and postprocessing (66%). The "typical" lead of a winning team is a computer scientist with a doctoral degree, five years of experience in biomedical image analysis, and four years of experience in deep learning. Two core general development strategies stood out for highly-ranked teams: the reflection of the metrics in the method design and the focus on analyzing and handling failure cases. According to the organizers, 43% of the winning algorithms exceeded the state of the art but only 11% completely solved the respective domain problem. The insights of our study could help researchers (1) improve algorithm development strategies when approaching new problems, and (2) focus on open research questions revealed by this work.
Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 7 of its 78 references that are in the map. First posted on arXiv 30 Mar 2023.
Builds on
7 of 78 references.
References · 7
- 2018Why rankings of biomedical image analysis competitions should be interpreted with carecited by 91
- 2019Methods and open-source toolkit for analyzing and visualizing challenge resultscited by 60
- 2021Common Limitations of Image Processing Metrics: A Picture Storycited by 59
- 2019BIAS: Transparent reporting of biomedical image analysis challengescited by 57
- 2022Assessing generalisability of deep learning-based polyp detection and segmentation methods through a computer vision challengecited by 27
- 2021Are we using appropriate segmentation metrics? Identifying correlates of human expert perception for CNN training beyond rolling the DICE coefficientcited by 26
- 2022Biomedical image analysis competitions: The state of current participation practicecited by 20
Built on by
Citing papers per year: 2023: 10; 2024: 4; 2025: 6; 2026: 1.
21 papers.
Sort:
By area:
- 2023AbdomenAtlas-8K: Annotating 8,000 CT Volumes for Multi-Organ Segmentation in Three Weekscited by 52
- 2023MedShapeNet -- A Large-Scale Dataset of 3D Medical Shapes for Computer Visioncited by 32
- 2023Domain Generalization in Computational Pathology: Survey and Guidelinescited by 29
- 2023Deep Interactive Segmentation of Medical Images: A Systematic Review and Taxonomycited by 27
- 2023Domain generalization across tumor types, laboratories, and species -- insights from the 2022 edition of the Mitosis Domain Generalization Challengecited by 25
- 2025How Well Do Supervised 3D Models Transfer to Medical Imaging Tasks?cited by 19
- 2023nnMobileNet: Rethinking CNN for Retinopathy Researchcited by 12
- 2025Surgical Visual Understanding (SurgVU) Datasetcited by 11
- 2024Multimodal Data Integration for Precision Oncology: Challenges and Future Directionscited by 10
- 2024A Review of Bayesian Uncertainty Quantification in Deep Probabilistic Image Segmentationcited by 6
- 2025Advances in Automated Fetal Brain MRI Segmentation and Biometry: Insights from the FeTA 2024 Challengecited by 5
- 2023MedAugment: Universal Automatic Data Augmentation Plug-in for Medical Image Analysiscited by 4
9 more
- 2025Leveraging Imperfection with MEDLEY A Multi-Model Approach Harnessing Bias in Medical AIcited by 4
- 2024Segmenting Medical Images with Limited Datacited by 3
- 2025ReX-MLE: The Autonomous Agent Benchmark for Medical Imaging Challengescited by 3
- 2023AutoPET Challenge 2023: Sliding Window-based Optimization of U-Netcited by 2
- 2024Beyond Knowledge Silos: Task Fingerprinting for Democratization of Medical Imaging AIcited by 2
- 2025Auditing Significance, Metric Choice, and Demographic Fairness in Medical AI Challengescited by 2
- 2023Improving Task Instructions for Data Annotators: How Clear Rules and Higher Pay Increase Performance in Data Annotation in the AI Economycited by 1
- 2023SMRVIS: Point cloud extraction from 3-D ultrasound for non-destructive testingcited by 0
- 2026Towards Global AI-Driven Cervical Cancer Screeningcited by 0
- 2026Towards Global AI-Driven Cervical Cancer Screening12 Jun 2026
- 2025Auditing Significance, Metric Choice, and Demographic Fairness in Medical AI Challenges22 Dec 2025
- 2025ReX-MLE: The Autonomous Agent Benchmark for Medical Imaging Challenges19 Dec 2025
- 2025Leveraging Imperfection with MEDLEY A Multi-Model Approach Harnessing Bias in Medical AI29 Aug 2025
- 2025Advances in Automated Fetal Brain MRI Segmentation and Biometry: Insights from the FeTA 2024 Challenge5 May 2025
- 2025How Well Do Supervised 3D Models Transfer to Medical Imaging Tasks?20 Jan 2025
- 2025Surgical Visual Understanding (SurgVU) Dataset16 Jan 2025
- 2024Beyond Knowledge Silos: Task Fingerprinting for Democratization of Medical Imaging AI11 Dec 2024
- 2024A Review of Bayesian Uncertainty Quantification in Deep Probabilistic Image Segmentation25 Nov 2024
- 2024Segmenting Medical Images with Limited Data12 Jul 2024
- 2024Multimodal Data Integration for Precision Oncology: Challenges and Future Directions28 Jun 2024
- 2023Improving Task Instructions for Data Annotators: How Clear Rules and Higher Pay Increase Performance in Data Annotation in the AI Economy22 Dec 2023
9 more
- 2023Deep Interactive Segmentation of Medical Images: A Systematic Review and Taxonomy23 Nov 2023
- 2023Domain Generalization in Computational Pathology: Survey and Guidelines30 Oct 2023
- 2023Domain generalization across tumor types, laboratories, and species -- insights from the 2022 edition of the Mitosis Domain Generalization Challenge27 Sep 2023
- 2023AutoPET Challenge 2023: Sliding Window-based Optimization of U-Net21 Sep 2023
- 2023MedShapeNet -- A Large-Scale Dataset of 3D Medical Shapes for Computer Vision30 Aug 2023
- 2023MedAugment: Universal Automatic Data Augmentation Plug-in for Medical Image Analysis30 Jun 2023
- 2023SMRVIS: Point cloud extraction from 3-D ultrasound for non-destructive testing7 Jun 2023
- 2023nnMobileNet: Rethinking CNN for Retinopathy Research2 Jun 2023
- 2023AbdomenAtlas-8K: Annotating 8,000 CT Volumes for Multi-Organ Segmentation in Three Weeks16 May 2023
- 2023AbdomenAtlas-8K: Annotating 8,000 CT Volumes for Multi-Organ Segmentation in Three Weekscited by 52
- 2023MedShapeNet -- A Large-Scale Dataset of 3D Medical Shapes for Computer Visioncited by 32
- 2023Domain Generalization in Computational Pathology: Survey and Guidelinescited by 29
- 2023Deep Interactive Segmentation of Medical Images: A Systematic Review and Taxonomycited by 27
- 2023Domain generalization across tumor types, laboratories, and species -- insights from the 2022 edition of the Mitosis Domain Generalization Challengecited by 25
- 2025How Well Do Supervised 3D Models Transfer to Medical Imaging Tasks?cited by 19
- 2023nnMobileNet: Rethinking CNN for Retinopathy Researchcited by 12
- 2025Surgical Visual Understanding (SurgVU) Datasetcited by 11
- 2024Multimodal Data Integration for Precision Oncology: Challenges and Future Directionscited by 10
- 2024A Review of Bayesian Uncertainty Quantification in Deep Probabilistic Image Segmentationcited by 6
- 2025Advances in Automated Fetal Brain MRI Segmentation and Biometry: Insights from the FeTA 2024 Challengecited by 5
- 2023MedAugment: Universal Automatic Data Augmentation Plug-in for Medical Image Analysiscited by 4
12 most cited of 20 All 20 (JSON)
- 2026Towards Global AI-Driven Cervical Cancer Screening12 Jun 2026
- 2025Auditing Significance, Metric Choice, and Demographic Fairness in Medical AI Challenges22 Dec 2025
- 2025Leveraging Imperfection with MEDLEY A Multi-Model Approach Harnessing Bias in Medical AI29 Aug 2025
- 2025Advances in Automated Fetal Brain MRI Segmentation and Biometry: Insights from the FeTA 2024 Challenge5 May 2025
- 2025How Well Do Supervised 3D Models Transfer to Medical Imaging Tasks?20 Jan 2025
- 2025Surgical Visual Understanding (SurgVU) Dataset16 Jan 2025
- 2024Beyond Knowledge Silos: Task Fingerprinting for Democratization of Medical Imaging AI11 Dec 2024
- 2024A Review of Bayesian Uncertainty Quantification in Deep Probabilistic Image Segmentation25 Nov 2024
- 2024Segmenting Medical Images with Limited Data12 Jul 2024
- 2024Multimodal Data Integration for Precision Oncology: Challenges and Future Directions28 Jun 2024
- 2023Improving Task Instructions for Data Annotators: How Clear Rules and Higher Pay Increase Performance in Data Annotation in the AI Economy22 Dec 2023
- 2023Deep Interactive Segmentation of Medical Images: A Systematic Review and Taxonomy23 Nov 2023
12 newest of 20 All 20 (JSON)