# Reproducibility in Machine Learning-Driven Research

arXiv: https://arxiv.org/abs/2307.10320 · HTML: https://mlsanity.com/paper/2307.10320/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 47 references that are in the map.
- First posted on arXiv 19 Jul 2023.

## Builds on (most cited first, 16 of 16)

- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)](https://mlsanity.com/paper/2003.12206/) (2020)
- [ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing](https://mlsanity.com/paper/2306.00622/) (2023)
- [Leakage and the Reproducibility Crisis in ML-based Science](https://mlsanity.com/paper/2207.07048/) (2022)
- [A Systematic Review of Reproducibility Research in Natural Language Processing](https://mlsanity.com/paper/2103.07929/) (2021)
- [Replicability Analysis for Natural Language Processing: Testing Significance with Multiple Datasets](https://mlsanity.com/paper/1709.09500/) (2017)
- [An introduction to Docker for reproducible research, with examples from the R environment](https://mlsanity.com/paper/1410.0846/) (2014)
- [Reproducibility in Machine Learning for Health](https://mlsanity.com/paper/1907.01463/) (2019)
- [The Unfairness of Popularity Bias in Music Recommendation: A Reproducibility Study](https://mlsanity.com/paper/1912.04696/) (2019)
- [Deterministic Implementations for Reproducibility in Deep Reinforcement Learning](https://mlsanity.com/paper/1809.05676/) (2018)
- [Validating Hyperspectral Image Segmentation](https://mlsanity.com/paper/1811.03707/) (2018)
- [Gym-Ignition: Reproducible Robotic Simulations for Reinforcement Learning](https://mlsanity.com/paper/1911.01715/) (2019)
- [Sustainable computational science: the ReScience initiative](https://mlsanity.com/paper/1707.04393/) (2017)
- [ReproducedPapers.org: Openly teaching and structuring machine learning reproducibility](https://mlsanity.com/paper/2012.01172/) (2020)
- [Reproducibility in machine learning for medical imaging](https://mlsanity.com/paper/2209.05097/) (2022)
- [Robustness of Meta Matrix Factorization Against Strict Privacy Constraints](https://mlsanity.com/paper/2101.06927/) (2021)

## Built on by (most cited first, 15 of 15)

- [Reproducibility in Machine Learning-based Research: Overview, Barriers and Drivers](https://mlsanity.com/paper/2406.14325/) (2024)
- [Verifiable evaluations of machine learning models using zkSNARKs](https://mlsanity.com/paper/2402.02675/) (2024)
- [Consistency in Language Models: Current Landscape, Challenges, and Future Directions](https://mlsanity.com/paper/2505.00268/) (2025)
- [Reflective Paper-to-Code Reproduction Enabled by Fine-Grained Verification](https://mlsanity.com/paper/2508.16671/) (2025)
- [The Impact of Feature Scaling In Machine Learning: Effects on Regression and Classification Tasks](https://mlsanity.com/paper/2506.08274/) (2025)
- [Improving the Reproducibility of Deep Learning Software: An Initial Investigation through a Case Study Analysis](https://mlsanity.com/paper/2505.03165/) (2025)
- [Confronting the Reproducibility Crisis: A Case Study of Challenges in Cybersecurity AI](https://mlsanity.com/paper/2405.18753/) (2024)
- [yProv4ML: Effortless Provenance Tracking for Machine Learning Systems](https://mlsanity.com/paper/2507.01078/) (2025)
- [PyHealth 2.0: A Comprehensive Open-Source Toolkit for Accessible and Reproducible Clinical Deep Learning](https://mlsanity.com/paper/2601.16414/) (2026)
- [PATH: A Discrete-sequence Dataset for Evaluating Online Unsupervised Anomaly Detection Approaches for Multivariate Time Series](https://mlsanity.com/paper/2411.13951/) (2024)
- [YAMLE: Yet Another Machine Learning Environment](https://mlsanity.com/paper/2402.06268/) (2024)
- [Practical Application and Limitations of AI Certification Catalogues in the Light of the AI Act](https://mlsanity.com/paper/2502.10398/) (2025)
- [Towards Reliable and Generalizable Differentially Private Machine Learning (Extended Version)](https://mlsanity.com/paper/2508.15141/) (2025)
- [SlideChain: Semantic Provenance for Lecture Understanding via Blockchain Registration](https://mlsanity.com/paper/2512.21684/) (2025)
- [Bridging the Reproducibility Divide: Open Source Software's Role in Standardizing Healthcare AI](https://mlsanity.com/paper/2603.03367/) (2026)

## Abstract (arXiv)

Research is facing a reproducibility crisis, in which the results and findings of many studies are difficult or even impossible to reproduce. This is also the case in machine learning (ML) and artificial intelligence (AI) research. Often, this is the case due to unpublished data and/or source-code, and due to sensitivity to ML training conditions. Although different solutions to address this issue are discussed in the research community such as using ML platforms, the level of reproducibility in ML-driven research is not increasing substantially. Therefore, in this mini survey, we review the literature on reproducibility in ML-driven research with three main aims: (i) reflect on the current situation of ML reproducibility in various research fields, (ii) identify reproducibility issues and barriers that exist in these research fields applying ML, and (iii) identify potential drivers such as tools, practices, and interventions that support ML reproducibility. With this, we hope to contribute to decisions on the viability of different solutions for supporting ML reproducibility.