# Exploring large scale public medical image datasets

arXiv: https://arxiv.org/abs/1907.12720 · HTML: https://mlsanity.com/paper/1907.12720/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Jul 2019.

## Built on by (most cited first, 30 of 37)

- [Exploring scalable medical image encoders beyond text supervision](https://mlsanity.com/paper/2401.10815/) (2024)
- [Deep Learning for Chest X-ray Analysis: A Survey](https://mlsanity.com/paper/2103.08700/) (2021)
- [Active label cleaning for improved dataset quality under resource constraints](https://mlsanity.com/paper/2109.00574/) (2021)
- [Deep Learning-based Patient Re-identification Is able to Exploit the Biometric Nature of Medical Chest X-ray Data](https://mlsanity.com/paper/2103.08562/) (2021)
- [How I failed machine learning in medical imaging -- shortcomings and recommendations](https://mlsanity.com/paper/2103.10292/) (2021)
- [VisualCheXbert: Addressing the Discrepancy Between Radiology Report Labels and Image Labels](https://mlsanity.com/paper/2102.11467/) (2021)
- [CyTran: A Cycle-Consistent Transformer with Multi-Level Consistency for Non-Contrast to Contrast CT Translation](https://mlsanity.com/paper/2110.06400/) (2021)
- [DICOM Imaging Router: An Open Deep Learning Framework for Classification of Body Parts from DICOM X-ray Scans](https://mlsanity.com/paper/2108.06490/) (2021)
- [Ground Truth Or Dare: Factors Affecting The Creation Of Medical Datasets For Training AI](https://mlsanity.com/paper/2309.12327/) (2023)
- [Rethinking Generalization: The Impact of Annotation Style on Medical Image Segmentation](https://mlsanity.com/paper/2210.17398/) (2022)
- [NVUM: Non-Volatile Unbiased Memory for Robust Medical Image Classification](https://mlsanity.com/paper/2103.04053/) (2021)
- [BoMD: Bag of Multi-label Descriptors for Noisy Chest X-ray Classification](https://mlsanity.com/paper/2203.01937/) (2022)
- [In the Picture: Medical Imaging Datasets, Artifacts, and their Living Review](https://mlsanity.com/paper/2501.10727/) (2025)
- [Responsible and Regulatory Conform Machine Learning for Medicine: A Survey of Challenges and Solutions](https://mlsanity.com/paper/2107.09546/) (2021)
- [An Accurate and Explainable Deep Learning System Improves Interobserver Agreement in the Interpretation of Chest Radiograph](https://mlsanity.com/paper/2208.03545/) (2022)
- [Nodule detection and generation on chest X-rays: NODE21 Challenge](https://mlsanity.com/paper/2401.02192/) (2024)
- [Lung-DDPM: Semantic Layout-guided Diffusion Models for Thoracic CT Image Synthesis](https://mlsanity.com/paper/2502.15204/) (2025)
- [Augmenting Chest X-ray Datasets with Non-Expert Annotations](https://mlsanity.com/paper/2309.02244/) (2023)
- [Copycats: the many lives of a publicly available medical imaging dataset](https://mlsanity.com/paper/2402.06353/) (2024)
- [Exploring Efficiency Frontiers of Thinking Budget in Medical Reasoning: Scaling Laws between Computational Resources and Reasoning Quality](https://mlsanity.com/paper/2508.12140/) (2025)
- [Did You Get What You Paid For? Rethinking Annotation Cost of Deep Learning Based Computer Aided Detection in Chest Radiographs](https://mlsanity.com/paper/2209.15314/) (2022)
- [On Evaluating Adversarial Robustness of Chest X-ray Classification: Pitfalls and Best Practices](https://mlsanity.com/paper/2212.08130/) (2022)
- [Limitations of Public Chest Radiography Datasets for Artificial Intelligence: Label Quality, Domain Shift, Bias and Evaluation Challenges](https://mlsanity.com/paper/2509.15107/) (2025)
- [Glaucoma Detection and Structured OCT Report Generation via a Fine-tuned Multimodal Large Language Model](https://mlsanity.com/paper/2510.02403/) (2025)
- [Data Shapley Value for Handling Noisy Labels: An application in Screening COVID-19 Pneumonia from Chest CT Scans](https://mlsanity.com/paper/2110.08726/) (2021)
- [Revisiting Performance Claims for Chest X-Ray Models Using Clinical Context](https://mlsanity.com/paper/2509.19671/) (2025)
- [Can Deep Learning Reliably Recognize Abnormality Patterns on Chest X-rays? A Multi-Reader Study Examining One Month of AI Implementation in Everyday Radiology Clinical Practice](https://mlsanity.com/paper/2305.10116/) (2023)
- [Automated Labeling of German Chest X-Ray Radiology Reports using Deep Learning](https://mlsanity.com/paper/2306.05997/) (2023)
- [Intuitions of Machine Learning Researchers about Transfer Learning for Medical Image Classification](https://mlsanity.com/paper/2510.00902/) (2025)
- [Curriculum Learning with Synthetic Data for Enhanced Pulmonary Nodule Detection in Chest Radiographs](https://mlsanity.com/paper/2510.07681/) (2025)

## Abstract (arXiv)

Rationale and Objectives: Medical artificial intelligence systems are dependent on well characterised large scale datasets. Recently released public datasets have been of great interest to the field, but pose specific challenges due to the disconnect they cause between data generation and data usage, potentially limiting the utility of these datasets. Materials and Methods: We visually explore two large public datasets, to determine how accurate the provided labels are and whether other subtle problems exist. The ChestXray14 dataset contains 112,120 frontal chest films, and the MURA dataset contains 40,561 upper limb radiographs. A subset of around 700 images from both datasets was reviewed by a board-certified radiologist, and the quality of the original labels was determined. Results: The ChestXray14 labels did not accurately reflect the visual content of the images, with positive predictive values mostly between 10% and 30% lower than the values presented in the original documentation. There were other significant problems, with examples of hidden stratification and label disambiguation failure. The MURA labels were more accurate, but the original normal/abnormal labels were inaccurate for the subset of cases with degenerative joint disease, with a sensitivity of 60% and a specificity of 82%. Conclusion: Visual inspection of images is a necessary component of understanding large image datasets. We recommend that teams producing public datasets should perform this important quality control procedure and include a thorough description of their findings, along with an explanation of the data generating procedures and labelling rules, in the documentation for their datasets.