mlsanityData to 7 Oct 2026

A Systematic Collection of Medical Image Datasets for Deep Learning

arXiv 2106.12864 ↗First posted 24 Jun 2021eess.IV cs.CV cs.LGcited by 29

Not on the map. Why?

Abstract (arXiv)

The astounding success made by artificial intelligence (AI) in healthcare and other fields proves that AI can achieve human-like performance. However, success always comes with challenges. Deep learning algorithms are data-dependent and require large datasets for training. The lack of data in the medical imaging field creates a bottleneck for the application of deep learning to medical image analysis. Medical image acquisition, annotation, and analysis are costly, and their usage is constrained by ethical restrictions. They also require many resources, such as human expertise and funding. That makes it difficult for non-medical researchers to have access to useful and large medical data. Thus, as comprehensive as possible, this paper provides a collection of medical image datasets with their associated challenges for deep learning research. We have collected information of around three hundred datasets and challenges mainly reported between 2013 and 2020 and categorized them into four categories: head & neck, chest & abdomen, pathology & blood, and ``others''. Our paper has three purposes: 1) to provide a most up to date and complete list that can be used as a universal reference to easily find the datasets for clinical image analysis, 2) to guide researchers on the methodology to test and evaluate their methods' performance and robustness on relevant datasets, 3) to provide a ``route'' to relevant algorithms for the relevant medical topics, and challenge leaderboards.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 69 of its 311 references that are in the map. First posted on arXiv 24 Jun 2021.

Builds on

69 of 311 references.

References · 69

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2017Mask R-CNNcited by 7,259
  6. 2016Densely Connected Convolutional Networkscited by 6,691
  7. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  8. 2014Going Deeper with Convolutionscited by 4,231
  9. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  10. 2018YOLOv3: An Incremental Improvementcited by 2,652
  11. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
  12. 2019A Comprehensive Survey on Transfer Learningcited by 1,154
38 more
  1. 2017ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseasescited by 969
  2. 2020Meta-Learning in Neural Networks: A Surveycited by 944
  3. 2019Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)cited by 657
  4. 2017Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC)cited by 549
  5. 2018Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challengecited by 508
  6. 2019The Liver Tumor Segmentation Benchmark (LiTS)cited by 504
  7. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 467
  8. 2018fastMRI: An Open Dataset and Benchmarks for Accelerated MRIcited by 464
  9. 2019Generalizing from a Few Examples: A Survey on Few-Shot Learningcited by 462
  10. 2018Rotation Equivariant CNNs for Digital Pathologycited by 420
  11. 2019A large annotated medical image dataset for the development and evaluation of segmentation algorithmscited by 371
  12. 2020CHAOS Challenge -- Combined (CT-MR) Healthy Abdominal Organ Segmentationcited by 307
  13. 2020COVID-Net: A Tailored Deep Convolutional Neural Network Design for Detection of COVID-19 Cases from Chest X-Ray Imagescited by 264
  14. 2019Deep Learning for Segmentation using an Open Large-Scale Dataset in 2D Echocardiographycited by 243
  15. 2019REFUGE Challenge: A Unified Framework for Evaluating Automated Methods for Glaucoma Assessment from Fundus Photographscited by 226
  16. 2020Can AI help in screening Viral and COVID-19 pneumonia?cited by 200
  17. 2016Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: the LUNA16 challengecited by 198
  18. 2020CORD-19: The COVID-19 Open Research Datasetcited by 179
  19. 2019The KiTS19 Challenge Data: 300 Kidney Tumor Cases with Clinical Context, CT Semantic Segmentations, and Surgical Outcomescited by 168
  20. 2015DeepOrgan: Multi-level Deep Convolutional Networks for Automated Pancreas Segmentationcited by 158
  21. 2018Multi-Institutional Deep Learning Modeling Without Sharing Patient Data: A Feasibility Study on Brain Tumor Segmentationcited by 148
  22. 2016Multivariate mixture model for myocardium segmentation combining multi-source imagescited by 133
  23. 2018BACH: Grand Challenge on Breast Cancer Histology Imagescited by 131
  24. 2015Generic decoding of seen and imagined objects using hierarchical visual featurescited by 119
  25. 2019Standardized Assessment of Automatic Segmentation of White Matter Hyperintensities and Results of the WMH Segmentation Challengecited by 115
  26. 2020VerSe: A Vertebrae Labelling and Segmentation Benchmark for Multi-detector CT Imagescited by 111
  27. 2020COVID-CT-Dataset: A CT Scan Dataset about COVID-19cited by 93
  28. 2014A New 2.5D Representation for Lymph Node Detection using Random Sets of Deep Convolutional Neural Network Observationscited by 56
  29. 2018Reconstructing Faces from fMRI Patterns using Deep Generative Neural Networkscited by 43
  30. 2018Few-shot 3D Multi-modal Medical Image Segmentation using Generative Adversarial Learningcited by 36
  31. 2017Automatic Brain Tumor Segmentation using Cascaded Anisotropic Convolutional Neural Networkscited by 33
  32. 2017Radiomics strategies for risk assessment of tumour failure in head-and-neck cancercited by 31
  33. 2019Intracranial Hemorrhage Segmentation Using Deep Convolutional Modelcited by 29
  34. 2020A Critic Evaluation of Methods for COVID-19 Automatic Detection from X-Ray Imagescited by 28
  35. 2020Multi-Site Infant Brain Segmentation Algorithms: The iSeg-2019 Challengecited by 28
  36. 2019Signet Ring Cell Detection With a Semi-supervised Learning Frameworkcited by 27
  37. 2020POCOVID-Net: Automatic Detection of COVID-19 From a New Lung Ultrasound Imaging Dataset (POCUS)cited by 25
  38. 2018Chest X-Ray Analysis of Tuberculosis by Deep Learning with Segmentation and Augmentationcited by 24

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 6; 2024: 12; 2025: 7; 2026: 3.

29 papers.

Sort:

By area:

  1. 2024CT2Rep: Automated Radiology Report Generation for 3D Medical Imagingcited by 96
  2. 2024VILA-M3: Enhancing Vision-Language Models with Medical Expert Knowledgecited by 50
  3. 2023Domain Generalization in Computational Pathology: Survey and Guidelinescited by 29
  4. 2025Primus: Enforcing Attention Usage for 3D Medical Image Segmentationcited by 22
  5. 2023Transformers in Small Object Detection: A Benchmark and Survey of State-of-the-Artcited by 15
  6. 20243D-CT-GPT: Generating 3D Radiology Reports through Integration of Large Vision-Language Modelscited by 15
  7. 2025In the Picture: Medical Imaging Datasets, Artifacts, and their Living Reviewcited by 8
  8. 2022Trichomonas Vaginalis Segmentation in Microscope Imagescited by 5
  9. 2024Copycats: the many lives of a publicly available medical imaging datasetcited by 5
  10. 2024Navigating Distribution Shifts in Medical Image Analysis: A Surveycited by 5
  11. 2024VIS-MAE: An Efficient Self-supervised Learning Approach on Medical Image Segmentation and Classificationcited by 4
  12. 2025IRS: Incremental Relationship-guided Segmentation for Digital Pathologycited by 2
17 more
  1. 2025A2LC: Active and Automated Label Correction for Semantic Segmentationcited by 2
  2. 2023Prototypical Cross-domain Knowledge Transfer for Cervical Dysplasia Visual Inspectioncited by 1
  3. 2024Multi-dimension Transformer with Attention-based Filtering for Medical Image Segmentationcited by 1
  4. 2024S-CycleGAN: Semantic Segmentation Enhanced CT-Ultrasound Image-to-Image Translation for Robotic Ultrasonographycited by 1
  5. 2025BI-RADS prediction of mammographic masses using uncertainty information extracted from a Bayesian Deep Learning modelcited by 1
  6. 2026Scaling Vision Language Models for Pharmaceutical Long Form Video Reasoning on Industrial GenAI Platformcited by 1
  7. 2023Towards Resource-Efficient Streaming of Large-Scale Medical Image Datasets for Deep Learningcited by 0
  8. 2023MS-UNet-v2: Adaptive Denoising Method and Training Strategy for Medical Image Segmentation with Small Training Datacited by 0
  9. 2023Resource-Limited Automated Ki67 Index Estimation in Breast Cancercited by 0
  10. 2024Additional Look into GAN-based Augmentation for Deep Learning COVID-19 Image Classificationcited by 0
  11. 2024PRECISe : Prototype-Reservation for Explainable Classification under Imbalanced and Scarce-Data Settingscited by 0
  12. 2024MozzaVID: Mozzarella Volumetric Image Datasetcited by 0
  13. 2024Review of Fruit Tree Image Segmentationcited by 0
  14. 2025HyTver: A Novel Loss Function for Longitudinal Multiple Sclerosis Lesion Segmentationcited by 0
  15. 2025Intuitions of Machine Learning Researchers about Transfer Learning for Medical Image Classificationcited by 0
  16. 2026Evaluating Few-Shot Pill Recognition Under Visual Domain Shiftcited by 0
  17. 2026Crossing the Validation Crisis: Cross-Validation Reduces Benchmarking Variance Surprisingly Wellcited by 0