# Reproducibility of the Methods in Medical Imaging with Deep Learning

arXiv: https://arxiv.org/abs/2210.11146 · HTML: https://mlsanity.com/paper/2210.11146/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 18 references that are in the map.
- First posted on arXiv 20 Oct 2022.

## Builds on (most cited first, 5 of 5)

- [Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)](https://mlsanity.com/paper/2003.12206/) (2020)
- [On the State of the Art of Evaluation in Neural Language Models](https://mlsanity.com/paper/1707.05589/) (2017)
- [A Step Toward Quantifying Independently Reproducible Machine Learning Research](https://mlsanity.com/paper/1909.06674/) (2019)
- [Active MR k-space Sampling with Reinforcement Learning](https://mlsanity.com/paper/2007.10469/) (2020)
- [The MICCAI Hackathon on reproducibility, diversity, and selection of papers at the MICCAI conference](https://mlsanity.com/paper/2103.05437/) (2021)

## Built on by (most cited first, 5 of 5)

- [Recent Methodological Advances in Federated Learning for Healthcare](https://mlsanity.com/paper/2310.02874/) (2023)
- [Copycats: the many lives of a publicly available medical imaging dataset](https://mlsanity.com/paper/2402.06353/) (2024)
- [[Citation needed] Data usage and citation practices in medical imaging conferences](https://mlsanity.com/paper/2402.03003/) (2024)
- [Blockchain-Enabled Privacy-Preserving Second-Order Federated Edge Learning in Personalized Healthcare](https://mlsanity.com/paper/2506.00416/) (2025)
- [Code Sharing In Prediction Model Research: A Scoping Review](https://mlsanity.com/paper/2604.06212/) (2026)

## Abstract (arXiv)

Concerns about the reproducibility of deep learning research are more prominent than ever, with no clear solution in sight. The relevance of machine learning research can only be improved if we also employ empirical rigor that incorporates reproducibility guidelines, especially so in the medical imaging field. The Medical Imaging with Deep Learning (MIDL) conference has made advancements in this direction by advocating open access, and recently also recommending authors to make their code public - both aspects being adopted by the majority of the conference submissions. This helps the reproducibility of the methods, however, there is currently little or no support for further evaluation of these supplementary material, making them vulnerable to poor quality, which affects the impact of the entire submission. We have evaluated all accepted full paper submissions to MIDL between 2018 and 2022 using established, but slightly adjusted guidelines on reproducibility and the quality of the public repositories. The evaluations show that publishing repositories and using public datasets are becoming more popular, which helps traceability, but the quality of the repositories has not improved over the years, leaving room for improvement in every aspect of designing repositories. Merely 22% of all submissions contain a repository that were deemed repeatable using our evaluations. From the commonly encountered issues during the evaluations, we propose a set of guidelines for machine learning-related research for medical imaging applications, adjusted specifically for future submissions to MIDL.