# Project Imaging-X: A Survey of 1000+ Open-Access Medical Imaging Datasets for Foundation Model Development arXiv: https://arxiv.org/abs/2603.27460 ยท HTML: https://mlsanity.com/paper/2603.27460/ ## Key facts - Cited by 2 papers in the map (data through 14 Oct 2026). - Builds on 101 of its 937 references that are in the map. - First posted on arXiv 29 Mar 2026. ## Builds on (most cited first, 30 of 101) - [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023) - [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024) - [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023) - [DINOv3](https://mlsanity.com/paper/2508.10104/) (2025) - [Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1902.03368/) (2019) - [Contrastive Learning of Medical Visual Representations from Paired Images and Text](https://mlsanity.com/paper/2010.00747/) (2020) - [The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classification](https://mlsanity.com/paper/2107.02314/) (2021) - [The Liver Tumor Segmentation Benchmark (LiTS)](https://mlsanity.com/paper/1901.04056/) (2019) - [PathVQA: 30000+ Questions for Medical Visual Question Answering](https://mlsanity.com/paper/2003.10286/) (2020) - [MedGemma Technical Report](https://mlsanity.com/paper/2507.05201/) (2025) - [Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomography](https://mlsanity.com/paper/2403.17834/) (2024) - [Skin Lesion Analysis toward Melanoma Detection: A Challenge at the International Symposium on Biomedical Imaging (ISBI) 2016, hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1605.01397/) (2016) - [The KiTS19 Challenge Data: 300 Kidney Tumor Cases with Clinical Context, CT Semantic Segmentations, and Surgical Outcomes](https://mlsanity.com/paper/1904.00445/) (2019) - [2017 Robotic Instrument Segmentation Challenge](https://mlsanity.com/paper/1902.06426/) (2019) - [M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Models](https://mlsanity.com/paper/2404.00578/) (2024) - [2018 Robotic Scene Segmentation Challenge](https://mlsanity.com/paper/2001.11190/) (2020) - [CholecSeg8k: A Semantic Segmentation Dataset for Laparoscopic Cholecystectomy Based on Cholec80](https://mlsanity.com/paper/2012.12453/) (2020) - [Stereo Correspondence and Reconstruction of Endoscopic Data Challenge](https://mlsanity.com/paper/2101.01133/) (2021) - [The KiTS21 Challenge: Automatic segmentation of kidneys, renal tumors, and renal cysts in corticomedullary-phase CT](https://mlsanity.com/paper/2307.01984/) (2023) - [COVID-19 Image Data Collection](https://mlsanity.com/paper/2003.11597/) (2020) - [STU-Net: Scalable and Transferable Medical Image Segmentation Models Empowered by Large-Scale Supervised Pre-training](https://mlsanity.com/paper/2304.06716/) (2023) - [COVID-CT-Dataset: A CT Scan Dataset about COVID-19](https://mlsanity.com/paper/2003.13865/) (2020) - [Lung and Colon Cancer Histopathological Image Dataset (LC25000)](https://mlsanity.com/paper/1912.12142/) (2019) - [BIMCV COVID-19+: a large annotated dataset of RX and CT images from COVID-19 patients](https://mlsanity.com/paper/2006.01174/) (2020) - [ROSE: A Retinal OCT-Angiography Vessel Segmentation Dataset and New Model](https://mlsanity.com/paper/2007.05201/) (2020) - [Comparative Validation of Machine Learning Algorithms for Surgical Workflow and Skill Analysis with the HeiChole Benchmark](https://mlsanity.com/paper/2109.14956/) (2021) - [SAR-RARP50: Segmentation of surgical instrumentation and Action Recognition on Robot-Assisted Radical Prostatectomy Challenge](https://mlsanity.com/paper/2401.00496/) (2023) - [SA-Med2D-20M Dataset: Segment Anything in 2D Medical Imaging with 20 Million masks](https://mlsanity.com/paper/2311.11969/) (2023) - [Intern-S1: A Scientific Multimodal Foundation Model](https://mlsanity.com/paper/2508.15763/) (2025) - [Self-Supervised Vision Transformers Learn Visual Concepts in Histopathology](https://mlsanity.com/paper/2203.00585/) (2022) ## Built on by (most cited first, 2 of 2) - [Probing, Fusion, and Trustworthiness: A Systematic Evaluation of Foundation Model Representations for Multimodal Cancer Analysis](https://mlsanity.com/paper/2606.17115/) (2026) - [Solve the Missing First Step: Can VLMs Standardize Raw Heterogeneous Medical Data?](https://mlsanity.com/paper/2607.04694/) (2026) ## Abstract (arXiv) Foundation models have demonstrated remarkable success across diverse domains and tasks, primarily due to the thrive of large-scale, diverse, and high-quality datasets. However, in the field of medical imaging, the curation and assembling of such medical datasets are highly challenging due to the reliance on clinical expertise and strict ethical and privacy constraints, resulting in a scarcity of large-scale unified medical datasets and hindering the development of powerful medical foundation models. In this work, we present the largest survey to date of medical image datasets, covering over 1,000 open-access datasets with a systematic catalog of their modalities, tasks, anatomies, annotations, limitations, and potential for integration. Our analysis exposes a landscape that is modest in scale, fragmented across narrowly scoped tasks, and unevenly distributed across organs and modalities, which in turn limits the utility of existing medical image datasets for developing versatile and robust medical foundation models. To turn fragmentation into scale, we propose a metadata-driven fusion paradigm (MDFP) that integrates public datasets with shared modalities or tasks, thereby transforming multiple small data silos into larger, more coherent resources. Building on MDFP, we release an interactive discovery portal that enables end-to-end, automated medical image dataset integration, and compile all surveyed datasets into a unified, structured table that clearly summarizes their key characteristics and provides reference links, offering the community an accessible and comprehensive repository. By charting the current terrain and offering a principled path to dataset consolidation, our survey provides a practical roadmap for scaling medical imaging corpora, supporting faster data discovery, more principled dataset creation, and more capable medical foundation models.