mlsanityData to 14 Oct 2026

Why rankings of biomedical image analysis competitions should be interpreted with care

arXiv 1806.02051 ↗First posted 6 Jun 2018cs.CVcited by 91

Not on the map. Why?

Abstract (arXiv)

International challenges have become the standard for validation of biomedical image analysis methods. Given their scientific impact, it is surprising that a critical analysis of common practices related to the organization of challenges has not yet been performed. In this paper, we present a comprehensive analysis of biomedical image analysis challenges conducted up to now. We demonstrate the importance of challenges and show that the lack of quality control has critical consequences. First, reproducibility and interpretation of the results is often hampered as only a fraction of relevant information is typically provided. Second, the rank of an algorithm is generally not robust to a number of variables such as the test data used for validation, the ranking scheme applied and the observers that make the reference annotations. To overcome these problems, we recommend best practice guidelines and define open research questions to be addressed in the future.

Key facts. Cited by 91 papers in the map (data through 14 Oct 2026). First posted on arXiv 6 Jun 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 16; 2022: 12; 2023: 21; 2024: 12; 2025: 14; 2026: 16.

91 papers.

Sort:

By area:

  1. 2021The Medical Segmentation Decathloncited by 628
  2. 2022Metrics reloaded: Recommendations for image analysis validationcited by 195
  3. 2022Medical Image Segmentation Review: The success of U-Netcited by 191
  4. 2023The KiTS21 Challenge: Automatic segmentation of kidneys, renal tumors, and renal cysts in corticomedullary-phase CTcited by 103
  5. 2021MedPerf: Open Benchmarking Platform for Medical Artificial Intelligence using Federated Evaluationcited by 88
  6. 2021nnDetection: A Self-configuring Method for Medical Object Detectioncited by 60
  7. 2021Comparative Validation of Machine Learning Algorithms for Surgical Workflow and Skill Analysis with the HeiChole Benchmarkcited by 60
  8. 2021Common Limitations of Image Processing Metrics: A Picture Storycited by 59
  9. 2022CrossMoDA 2021 challenge: Benchmark of Cross-Modality Domain Adaptation techniques for Vestibular Schwannoma and Cochlea Segmentationcited by 58
  10. 2023Understanding metric-related pitfalls in image analysis validationcited by 58
  11. 2023SAR-RARP50: Segmentation of surgical instrumentation and Action Recognition on Robot-Assisted Radical Prostatectomy Challengecited by 56
  12. 2023Unleashing the Strengths of Unlabeled Data in Pan-cancer Abdominal Organ Quantification: the FLARE22 Challengecited by 50
38 more
  1. 2023The Multi-modality Cell Segmentation Challenge: Towards Universal Solutionscited by 39
  2. 2021How I failed machine learning in medical imaging -- shortcomings and recommendationscited by 37
  3. 2022Overview of the HECKTOR Challenge at MICCAI 2021: Automatic Head and Neck Tumor Segmentation and Outcome Prediction in PET/CT Imagescited by 30
  4. 2021MIcro-Surgical Anastomose Workflow recognition challenge reportcited by 28
  5. 2021Are we using appropriate segmentation metrics? Identifying correlates of human expert perception for CNN training beyond rolling the DICE coefficientcited by 26
  6. 2023Why is the winner the best?cited by 21
  7. 2022Labeling instructions matter in biomedical image analysiscited by 20
  8. 2022Inflation of test accuracy due to data leakage in deep learning-based classification of OCT imagescited by 17
  9. 2023Self-Supervised Pre-Training with Contrastive and Masked Autoencoder Methods for Dealing with Small Datasets in Deep Learning for Medical Imagingcited by 16
  10. 2024Navigating the Maze of Explainable AI: A Systematic Approach to Evaluating Methods and Metricscited by 16
  11. 2021Challenges for machine learning in clinical translation of big data imaging studiescited by 15
  12. 2024Deep Rib Fracture Instance Segmentation and Classification from CT on the RibFrac Challengecited by 14
  13. 2024Confidence intervals uncovered: Are we ready for real-world medical imaging AI?cited by 12
  14. 2025Beyond the LUMIR challenge: The pathway to foundational registration modelscited by 12
  15. 2023Why is plausibility surprisingly problematic as an XAI criterion?cited by 9
  16. 2021PointNu-Net: Keypoint-assisted Convolutional Neural Network for Simultaneous Multi-tissue Histology Nuclei Segmentation and Classificationcited by 8
  17. 2022Approaching Peak Ground Truthcited by 8
  18. 2023The Brain Tumor Segmentation (BraTS) Challenge: Local Synthesis of Healthy Brain Tissue via Inpaintingcited by 8
  19. 2024A Robust Ensemble Algorithm for Ischemic Stroke Lesion Segmentation: Generalizability and Clinical Utility Beyond the ISLES Challengecited by 8
  20. 2023Evaluating AI systems under uncertain ground truth: a case study in dermatologycited by 7
  21. 2022PEg TRAnsfer Workflow recognition challenge report: Does multi-modal data improve recognition?cited by 6
  22. 2022Deep Quality Estimation: Creating Surrogate Models for Human Quality Ratingscited by 6
  23. 2024LNQ 2023 challenge: Benchmark of weakly-supervised techniques for mediastinal lymph node quantificationcited by 6
  24. 2021How can we learn (more) from challenges? A statistical approach to driving future algorithm developmentcited by 5
  25. 2021Ten years of image analysis and machine learning competitions in dementiacited by 5
  26. 2025MedNeXt-v2: Scaling 3D ConvNeXts for Large-Scale Supervised Representation Learning in Medical Image Segmentationcited by 5
  27. 2021A new baseline for retinal vessel segmentation: Numerical identification and correction of methodological inconsistencies affecting 100+ paperscited by 4
  28. 2023The NCI Imaging Data Commons as a platform for reproducible research in computational pathologycited by 4
  29. 2023Deployment of Image Analysis Algorithms under Prevalence Shiftscited by 4
  30. 2023APIS: A paired CT-MRI dataset for ischemic stroke segmentation challengecited by 4
  31. 2025TUS-REC2024: A Challenge to Reconstruct 3D Freehand Ultrasound Without External Trackercited by 4
  32. 2023Efficient Large Scale Medical Image Dataset Preparation for Machine Learning Applicationscited by 3
  33. 2024Understanding implementation pitfalls of distance-based metrics for image segmentationcited by 3
  34. 2024MRGen: Segmentation Data Engine for Underrepresented MRI Modalitiescited by 3
  35. 2024Foundations of the Theory of Performance-Based Rankingcited by 3
  36. 2024The Tile: A 2D Map of Ranking Scores for Two-Class Classificationcited by 3
  37. 2026Towards Brain MRI Foundation Models for the Clinic: Findings from the FOMO25 Challengecited by 3
  38. 2021MixMicrobleedNet: segmentation of cerebral microbleeds using nnU-Netcited by 2

50 most cited of 91 All 91 (JSON)