mlsanityData to 7 Oct 2026

How I failed machine learning in medical imaging -- shortcomings and recommendations

arXiv 2103.10292 ↗First posted 18 Mar 2021eess.IV cs.CV cs.LGcited by 37

Not on the map. Why?

Abstract (arXiv)

Medical imaging is an important research field with many opportunities for improving patients' health. However, there are a number of challenges that are slowing down the progress of the field as a whole, such optimizing for publication. In this paper we reviewed several problems related to choosing datasets, methods, evaluation metrics, and publication strategies. With a review of literature and our own analysis, we show that at every step, potential biases can creep in. On a positive note, we also see that initiatives to counteract these problems are already being started. Finally we provide a broad range of recommendations on how to further these address problems in the future. For reproducibility, data and code for our analyses are available on urlhttps://github.com/GaelVaroquaux/mlmedimagingfailures

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Builds on 39 of its 122 references that are in the map. First posted on arXiv 18 Mar 2021.

Builds on

39 of 122 references.

References · 39

  1. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  2. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  3. 2018Datasheets for Datasetscited by 1,371
  4. 2017A Survey on Deep Learning in Medical Image Analysiscited by 1,182
  5. 2018Model Cards for Model Reportingcited by 1,068
  6. 2018Confounding variables can degrade generalization performance of radiological deep learning modelscited by 374
  7. 2019A large annotated medical image dataset for the development and evaluation of segmentation algorithmscited by 371
  8. 2020Are we done with ImageNet?cited by 329
  9. 2020A Metric Learning Reality Checkcited by 317
  10. 2019Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNetcited by 260
  11. 2020Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learningcited by 246
  12. 2018Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shiftcited by 240
27 more
  1. 2020A review of deep learning in medical imaging: Imaging traits, technology trends, case studies with progress highlights, and future promisescited by 230
  2. 2019Hidden Stratification Causes Clinically Meaningful Failures in Machine Learning for Medical Imagingcited by 215
  3. 2018Deep Learning: A Critical Appraisalcited by 214
  4. 2021Accounting for Variance in Machine Learning Benchmarkscited by 195
  5. 2019Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approachescited by 181
  6. 2018Not-so-supervised: a survey of semi-supervised, multi-instance, and transfer learning in medical image analysiscited by 178
  7. 2020Carbontracker: Tracking and Predicting the Carbon Footprint of Training Deep Learning Modelscited by 168
  8. 2020Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scanscited by 158
  9. 2015Should we really use post-hoc tests based on mean-ranks?cited by 116
  10. 2018Troubling Trends in Machine Learning Scholarshipcited by 107
  11. 2019Convolutional Neural Networks for Classification of Alzheimer's Disease: Overview and Reproducible Evaluationcited by 106
  12. 2018Why rankings of biomedical image analysis competitions should be interpreted with carecited by 90
  13. 2020On the Value of Out-of-Distribution Testing: An Example of Goodhart's Lawcited by 74
  14. 2019Can we trust deep learning models diagnosis? The impact of domain shift in chest radiograph classificationcited by 57
  15. 2020Detect and Correct Bias in Multi-Site Neuroimaging Datasetscited by 44
  16. 2017Cross-validation failure: small sample sizes lead to large error barscited by 41
  17. 2019Exploring large scale public medical image datasetscited by 37
  18. 2019Reproducibility in Machine Learning for Healthcited by 24
  19. 2020Risk of Training Diagnostic Algorithms on Data with Demographic Biascited by 23
  20. 2019A Survey of Crowdsourcing in Medical Image Analysiscited by 20
  21. 2018Replication study: Development and validation of deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographscited by 17
  22. 2019The Scientific Method in the Science of Machine Learningcited by 15
  23. 2019HARK Side of Deep Learning -- From Grad Student Descent to Automated Machine Learningcited by 13
  24. 2020Evaluating Progress on Machine Learning for Longitudinal Electronic Healthcare Datacited by 12
  25. 2018Learning to Unlearn: Building Immunity to Dataset Bias in Medical Imaging Studiescited by 7
  26. 2017Negative Results in Computer Vision: A Perspectivecited by 5
  27. 2018Detect, Quantify, and Incorporate Dataset Bias: A Neuroimaging Analysis on 12,207 Individualscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 4; 2025: 4; 2026: 26.

37 papers.

Sort:

By area:

  1. 2023Generative models improve fairness of medical classifiers under distribution shiftscited by 90
  2. 2023Artificial intelligence in digital pathology: a systematic review and meta-analysis of diagnostic test accuracycited by 9
  3. 2024Ctrl-GenAug: Controllable Generative Augmentation for Medical Sequence Classificationcited by 4
  4. 2024Evaluating Synthetic Data Generation for Domain Generalization in Fetal Brain MRI Segmentationcited by 3
  5. 2026MMBU: A Massive Multi-modal Biomedical Understanding Benchmark to Probe the Perception Capabilities of Vision-Language Modelscited by 3
  6. 2023Inventing a coin-flip classifier: Accounting for multiplicity in machine learning benchmark performancecited by 2
  7. 2024Measuring Model-Induced Discrimination via Efficient Fairness Approximationcited by 2
  8. 2025How many measurements are enough? Bayesian recovery in inverse problems with general distributionscited by 2
  9. 2026Vision-language models for chest radiography do not always need the imagecited by 2
  10. 2025State of Abdominal CT Datasets: A Critical Review of Bias, Clinical Relevance, and Real-world Applicabilitycited by 1
  11. 2025Medical Imaging AI Competitions Lack Fairnesscited by 1
  12. 2026Physics from Video: Identifiability of Time-Invariant Second-Order ODEs under Minimal Trajectory Conditionscited by 1
25 more
  1. 2026A Leakage-Aware Comparative Benchmark of Machine Learning, Deep Learning, and Transformer Models for Reliable Leukemia Detectioncited by 1
  2. 2024Twin: Tuning Learning Rate and Weight Decay of Deep Homogeneous Classifiers without Validationcited by 0
  3. 2025Predicting Brain Morphogenesis via Physics-Transfer Learningcited by 0
  4. 2026GP-VM×SMA: Benchmarking General-Purpose Vision Models and Specialized Architectures for 2D Medical Image Segmentationcited by 0
  5. 2026JDCNet: Confidence-Gated Privileged-Modality Distillation for Cost-Preserving X-ray Inferencecited by 0
  6. 2026Not All Forgetting Is Equal: Retention Dynamics in Fine-Tuned Image Classifierscited by 0
  7. 2026On Hallucinations in Inverse Problems: Fundamental Limits and Provable Assessment Methodscited by 0
  8. 2026MicroscopyMatching: Towards a Ready-to-use Framework for Microscopy Image Analysis in Diverse Conditionscited by 0
  9. 2026Which Anatomy Matters Under Limited Labels? A Data-Efficient Anatomy-Aware Benchmark for Cardiac Pathology Predictioncited by 0
  10. 2026Feynman Kac Reweighted Schr"odinger Bridge Matching for Surface-Based Tau PET Harmonizationcited by 0
  11. 2026When, Where, and How: Adaptive Binning for Tabular Self-Supervised Learningcited by 0
  12. 2026Agentic AI in medicine: architectures, applications, evaluation, and challenges for clinical translationcited by 0
  13. 2026RadPRISM: Schema-stratified radiology-report supervision for concept-disentangled image representations and visual groundingcited by 0
  14. 2026Similarity Weighted Aggregation with Global Differential Privacy for Federated Brain Lesion Segmentationcited by 0
  15. 2026When Oracle Conditioning Misleads Deployment: Conditioning-Availability Bias in Echocardiographic Segmentationcited by 0
  16. 2026B-MIM: Biased Masked Image Modeling for Generalizable Segmentation of Fine-Grained Anatomical Structurescited by 0
  17. 2026MedSegBenchmarker: A Raw-Count-First Framework for Controlled 2D Medical Image Segmentation Benchmarkscited by 0
  18. 2026Self-supervised Pre-training Helps Retinal Disease Progression Modelling Most When Data Is Scarcecited by 0
  19. 2026TRACE: Tractable Routing Autoencoder for Clinical ECGcited by 0
  20. 2026Leakage-Safe Machine Learning for Hydrogen Embrittlement Detection in 316L Stainless Steel: A Region-Held-Out Evaluation of Texture and Deep Features in SEM Micrographscited by 0
  21. 2026M3-Score: Fidelity, Memorization and Coverage as Separate Axes for Evaluating Generative Radiology Image Modelscited by 0
  22. 2026Evaluation Choices Shape Biomedical ML Claims: A Pediatric Pneumonia Benchmark Case Studycited by 0
  23. 2026Does Gradient Conflict Predict the Understanding--Generation Trade-off? A Controlled Audit of Conflict-Metric Validity in Unified Multimodal Modelscited by 0
  24. 2026Scores That Hold, Benchmarks That Leak: Measuring Dataset Contamination in Public Brain-Tumor MRI Classificationcited by 0
  25. 2026MIRTO: a registration-gated, multiverse-tested evaluation protocol for unsupervised anomaly segmentation in brain MRIcited by 0