mlsanityData to 7 Oct 2026

The Medical Segmentation Decathlon

arXiv 2106.05735 ↗First posted 10 Jun 2021eess.IV cs.CV cs.LGcited by 620

Not on the map. Why?

Abstract (arXiv)

International challenges have become the de facto standard for comparative assessment of image analysis algorithms given a specific task. Segmentation is so far the most widely investigated medical image processing task, but the various segmentation challenges have typically been organized in isolation, such that algorithm development was driven by the need to tackle a single specific clinical problem. We hypothesized that a method capable of performing well on multiple tasks will generalize well to a previously unseen task and potentially outperform a custom-designed solution. To investigate the hypothesis, we organized the Medical Segmentation Decathlon (MSD) - a biomedical image analysis challenge, in which algorithms compete in a multitude of both tasks and modalities. The underlying data set was designed to explore the axis of difficulties typically encountered when dealing with medical images, such as small data sets, unbalanced labels, multi-site data and small objects. The MSD challenge confirmed that algorithms with a consistent good performance on a set of tasks preserved their good average performance on a different set of previously unseen tasks. Moreover, by monitoring the MSD winner for two years, we found that this algorithm continued generalizing well to a wide range of other clinical problems, further confirming our hypothesis. Three main conclusions can be drawn from this study: (1) state-of-the-art image segmentation algorithms are mature, accurate, and generalize well when retrained on unseen tasks; (2) consistent algorithmic performance across multiple tasks is a strong surrogate of algorithmic generalizability; (3) the training of accurate AI segmentation models is now commoditized to non AI experts.

Key facts. Cited by 620 papers in the map (data through 7 Oct 2026). Builds on 22 of its 51 references that are in the map. First posted on arXiv 10 Jun 2021.

Builds on

22 of 51 references.

References · 22

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2016V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentationcited by 2,284
  5. 2017Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC)cited by 549
  6. 2018Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challengecited by 508
  7. 2019The Liver Tumor Segmentation Benchmark (LiTS)cited by 504
  8. 2019AutoML: A Survey of the State-of-the-Artcited by 388
  9. 2019A large annotated medical image dataset for the development and evaluation of segmentation algorithmscited by 371
  10. 2016Gland Segmentation in Colon Histology Images: The GlaS Challenge Contestcited by 203
  11. 2021Revisiting ResNets: Improved Training and Scaling Strategiescited by 158
  12. 2018Why rankings of biomedical image analysis competitions should be interpreted with carecited by 90
10 more
  1. 2018Deep learning to achieve clinically applicable segmentation of head and neck anatomy for radiotherapycited by 76
  2. 2017Predicting Cardiovascular Risk Factors from Retinal Fundus Photographs using Deep Learningcited by 63
  3. 2019Methods and open-source toolkit for analyzing and visualizing challenge resultscited by 60
  4. 2021Common Limitations of Image Processing Metrics: A Picture Storycited by 58
  5. 20183D Semi-Supervised Learning with Uncertainty-Aware Multi-View Co-Trainingcited by 57
  6. 2019BIAS: Transparent reporting of biomedical image analysis challengescited by 57
  7. 2021DiNTS: Differentiable Neural Network Topology Search for 3D Medical Image Segmentationcited by 47
  8. 2019One Network to Segment Them All: A General, Lightweight System for Accurate 3D Medical Image Segmentationcited by 24
  9. 2021Cutting-edge 3D Medical Image Segmentation Methods in 2020: Are Happy Families All Alike?cited by 7
  10. 2020AutoML Segmentation for 3D Medical Image Data: Contribution to the MSD Challenge 2018cited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 25; 2022: 55; 2023: 148; 2024: 152; 2025: 153; 2026: 87.

620 papers.

Sort:

By area:

  1. 2022Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Imagescited by 823
  2. 2021MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classificationcited by 682
  3. 2022MONAI: An open-source framework for deep learning in healthcarecited by 438
  4. 2021Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysiscited by 418
  5. 2022AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentationcited by 347
  6. 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 333
  7. 2024nnU-Net Revisited: A Call for Rigorous Validation in 3D Medical Image Segmentationcited by 235
  8. 2023Segment Anything in Medical Imagescited by 226
  9. 2023CLIP-Driven Universal Model for Organ Segmentation and Tumor Detectioncited by 224
  10. 2024Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomographycited by 192
  11. 2022Metrics reloaded: Recommendations for image analysis validationcited by 190
  12. 2022Medical Image Segmentation Review: The success of U-Netcited by 190
38 more
  1. 2021nnFormer: Interleaved Transformer for Volumetric Segmentationcited by 179
  2. 2023On the Challenges and Perspectives of Foundation Models for Medical Image Analysiscited by 135
  3. 2021Learn2Reg: comprehensive multi-task medical image registration challenge, dataset and evaluation in the era of deep learningcited by 125
  4. 2021Causality-inspired Single-source Domain Generalization for Medical Image Segmentationcited by 118
  5. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  6. 2024Segment Anything Model for Medical Image Segmentation: Current Applications and Future Directionscited by 97
  7. 2023STU-Net: Scalable and Transferable Medical Image Segmentation Models Empowered by Large-Scale Supervised Pre-trainingcited by 96
  8. 2024MAISI: Medical AI for Synthetic Imagingcited by 86
  9. 2023MA-SAM: Modality-agnostic SAM Adaptation for 3D Medical Image Segmentationcited by 82
  10. 2023SAM Struggles in Concealed Scenes -- Empirical Study on Segment Anythingcited by 75
  11. 2022Learning with Limited Annotations: A Survey on Deep Semi-Supervised Learning for Medical Image Segmentationcited by 74
  12. 20233DSAM-adapter: Holistic adaptation of SAM from 2D to 3D for promptable tumor segmentationcited by 70
  13. 2022Self Pre-training with Masked Autoencoders for Medical Image Classification and Segmentationcited by 69
  14. 2024A Survey on Visual Anomaly Detection: Challenge, Approach, and Prospectcited by 69
  15. 2023Generative AI for Medical Imaging: extending the MONAI Frameworkcited by 68
  16. 2024Towards Generalizable Tumor Synthesiscited by 65
  17. 2024AbdomenAtlas: A Large-Scale, Detailed-Annotated, & Multi-Center Dataset for Efficient Transfer Learning and Open Algorithmic Benchmarkingcited by 64
  18. 2024VISTA3D: A Unified Segmentation Foundation Model For 3D Medical Imagingcited by 62
  19. 2023Large AI Models in Health Informatics: Applications, Challenges, and the Futurecited by 61
  20. 2023Large-Vocabulary Segmentation for Medical Images with Text Promptscited by 60
  21. 2022CrossMoDA 2021 challenge: Benchmark of Cross-Modality Domain Adaptation techniques for Vestibular Schwannoma and Cochlea Segmentationcited by 58
  22. 2023Label-Free Liver Tumor Segmentationcited by 56
  23. 2023SA-Med2D-20M Dataset: Segment Anything in 2D Medical Imaging with 20 Million maskscited by 54
  24. 2023AbdomenAtlas-8K: Annotating 8,000 CT Volumes for Multi-Organ Segmentation in Three Weekscited by 52
  25. 2023Unleashing the Strengths of Unlabeled Data in Pan-cancer Abdominal Organ Quantification: the FLARE22 Challengecited by 50
  26. 2023Generative Text-Guided 3D Vision-Language Pretraining for Unified Medical Image Segmentationcited by 47
  27. 2023UniSeg: A Prompt-driven Universal Segmentation Model as well as A Strong Representation Learnercited by 46
  28. 2022A Data-scalable Transformer for Medical Image Segmentation: Architecture, Model Efficiency, and Benchmarkcited by 45
  29. 2023MultiTalent: A Multi-Dataset Approach to Medical Image Segmentationcited by 45
  30. 2023Computer-Vision Benchmark Segment-Anything Model (SAM) in Medical Images: Accuracy in 12 Datasetscited by 45
  31. 2024UltraLight VM-UNet: Parallel Vision Mamba Significantly Reduces Parameters for Skin Lesion Segmentationcited by 45
  32. 2024MedImageInsight: An Open-Source Embedding Model for General Domain Medical Imagingcited by 45
  33. 2023Feature Alignment and Uniformity for Test Time Adaptationcited by 42
  34. 2021A Robust Volumetric Transformer for Accurate 3D Tumor Segmentationcited by 41
  35. 2022Closing the Generalization Gap of Cross-silo Federated Medical Image Segmentationcited by 41
  36. 2023Diff-UNet: A Diffusion Embedded Network for Volumetric Segmentationcited by 41
  37. 2021Optimized U-Net for Brain Tumor Segmentationcited by 40
  38. 2023Overcoming Data Scarcity in Biomedical Imaging with a Foundational Multi-Task Modelcited by 40

50 most cited of 620 All 620 (JSON)