# Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans

arXiv: https://arxiv.org/abs/2008.06388 · HTML: https://mlsanity.com/paper/2008.06388/

## Key facts

- Cited by 158 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Aug 2020.

## Built on by (most cited first, 30 of 158)

- [Fishr: Invariant Gradient Variances for Out-of-Distribution Generalization](https://mlsanity.com/paper/2109.02934/) (2021)
- [Recent advances and clinical applications of deep learning in medical image analysis](https://mlsanity.com/paper/2105.13381/) (2021)
- [Weak baselines and reporting biases lead to overoptimism in machine learning for fluid-related partial differential equations](https://mlsanity.com/paper/2407.07218/) (2024)
- [The Fallacy of AI Functionality](https://mlsanity.com/paper/2206.09511/) (2022)
- [Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies](https://mlsanity.com/paper/2112.11471/) (2021)
- [Unbox the Black-box for the Medical Explainable AI via Multi-modal and Multi-centre Data Fusion: A Mini-Review, Two Showcases and Beyond](https://mlsanity.com/paper/2102.01998/) (2021)
- [Leakage and the Reproducibility Crisis in ML-based Science](https://mlsanity.com/paper/2207.07048/) (2022)
- [Robust and Efficient Medical Imaging with Self-Supervision](https://mlsanity.com/paper/2205.09723/) (2022)
- [How I failed machine learning in medical imaging -- shortcomings and recommendations](https://mlsanity.com/paper/2103.10292/) (2021)
- [How to avoid machine learning pitfalls: a guide for academic researchers](https://mlsanity.com/paper/2108.02497/) (2021)
- [Sources of Irreproducibility in Machine Learning: A Review](https://mlsanity.com/paper/2204.07610/) (2022)
- [RadEdit: stress-testing biomedical vision models via diffusion image editing](https://mlsanity.com/paper/2312.12865/) (2023)
- [Classification of datasets with imputed missing values: does imputation quality matter?](https://mlsanity.com/paper/2206.08478/) (2022)
- [Vision Transformer using Low-level Chest X-ray Feature Corpus for COVID-19 Diagnosis and Severity Quantification](https://mlsanity.com/paper/2104.07235/) (2021)
- [Sounds of COVID-19: exploring realistic performance of audio-based digital testing](https://mlsanity.com/paper/2106.15523/) (2021)
- [The Pitfalls of Memorization: When Memorization Hurts Generalization](https://mlsanity.com/paper/2412.07684/) (2024)
- [Federated Split Vision Transformer for COVID-19 CXR Diagnosis using Task-Agnostic Training](https://mlsanity.com/paper/2111.01338/) (2021)
- [I Find Your Lack of Uncertainty in Computer Vision Disturbing](https://mlsanity.com/paper/2104.08188/) (2021)
- [Multimodal Explainability via Latent Shift applied to COVID-19 stratification](https://mlsanity.com/paper/2212.14084/) (2022)
- [DC-Check: A Data-Centric AI checklist to guide the development of reliable machine learning systems](https://mlsanity.com/paper/2211.05764/) (2022)
- [REFORMS: Reporting Standards for Machine Learning Based Science](https://mlsanity.com/paper/2308.07832/) (2023)
- [Generalizability of Machine Learning Models: Quantitative Evaluation of Three Methodological Pitfalls](https://mlsanity.com/paper/2202.01337/) (2022)
- [Audio-based AI classifiers show no evidence of improved COVID-19 screening over simple symptoms checkers](https://mlsanity.com/paper/2212.08570/) (2022)
- [Don't Push the Button! Exploring Data Leakage Risks in Machine Learning and Transfer Learning](https://mlsanity.com/paper/2401.13796/) (2024)
- [Explainable AI For COVID-19 CT Classifiers: An Initial Comparison Study](https://mlsanity.com/paper/2104.14506/) (2021)
- [Boundary Guided Semantic Learning for Real-time COVID-19 Lung Infection Segmentation System](https://mlsanity.com/paper/2209.02934/) (2022)
- [Recent Methodological Advances in Federated Learning for Healthcare](https://mlsanity.com/paper/2310.02874/) (2023)
- [AutoPrognosis 2.0: Democratizing Diagnostic and Prognostic Modeling in Healthcare with Automated Machine Learning](https://mlsanity.com/paper/2210.12090/) (2022)
- [Preemptively Pruning Clever-Hans Strategies in Deep Neural Networks](https://mlsanity.com/paper/2304.05727/) (2023)
- [Artificial intelligence in digital pathology: a systematic review and meta-analysis of diagnostic test accuracy](https://mlsanity.com/paper/2306.07999/) (2023)

## Abstract (arXiv)

Machine learning methods offer great promise for fast and accurate detection and prognostication of COVID-19 from standard-of-care chest radiographs (CXR) and computed tomography (CT) images. Many articles have been published in 2020 describing new machine learning-based models for both of these tasks, but it is unclear which are of potential clinical utility. In this systematic review, we search EMBASE via OVID, MEDLINE via PubMed, bioRxiv, medRxiv and arXiv for published papers and preprints uploaded from January 1, 2020 to October 3, 2020 which describe new machine learning models for the diagnosis or prognosis of COVID-19 from CXR or CT images. Our search identified 2,212 studies, of which 415 were included after initial screening and, after quality screening, 61 studies were included in this systematic review. Our review finds that none of the models identified are of potential clinical use due to methodological flaws and/or underlying biases. This is a major weakness, given the urgency with which validated COVID-19 models are needed. To address this, we give many recommendations which, if followed, will solve these issues and lead to higher quality model development and well documented manuscripts.