# What is Reproducibility in Artificial Intelligence and Machine Learning Research?

arXiv: https://arxiv.org/abs/2407.10239 · HTML: https://mlsanity.com/paper/2407.10239/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 41 references that are in the map.
- First posted on arXiv 29 Apr 2024.

## Builds on (most cited first, 8 of 8)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [Confounding variables can degrade generalization performance of radiological deep learning models](https://mlsanity.com/paper/1807.00431/) (2018)
- [Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control](https://mlsanity.com/paper/1708.04133/) (2017)
- [A Step Toward Quantifying Independently Reproducible Machine Learning Research](https://mlsanity.com/paper/1909.06674/) (2019)
- [The Fundamental Principles of Reproducibility](https://mlsanity.com/paper/2011.10098/) (2020)
- [Terminologies for Reproducible Research](https://mlsanity.com/paper/1802.03311/) (2018)
- [A Survey on Reproducibility in Parallel Computing](https://mlsanity.com/paper/1511.04217/) (2015)

## Built on by (most cited first, 11 of 11)

- [The Story is Not the Science: Execution-Grounded Evaluation of Mechanistic Interpretability Research](https://mlsanity.com/paper/2602.18458/) (2026)
- [Reproducibility: The New Frontier in AI Governance](https://mlsanity.com/paper/2510.11595/) (2025)
- [Open and Sustainable AI: challenges, opportunities and the road ahead in the life sciences (October 2025 -- Version 2)](https://mlsanity.com/paper/2505.16619/) (2025)
- [Assessing Reproducibility in Evolutionary Computation: A Case Study using Human- and LLM-based Assessment](https://mlsanity.com/paper/2602.07059/) (2026)
- ["ScatSpotter" -- A Dog Poop Detection Dataset](https://mlsanity.com/paper/2412.16473/) (2024)
- [Sensitivity of Stability: Theoretical & Empirical Analysis of Replicability for Adaptive Data Selection in Transfer Learning](https://mlsanity.com/paper/2508.04901/) (2025)
- [A K-Means, Ward and DBSCAN repeatability study](https://mlsanity.com/paper/2512.19772/) (2025)
- [Bridging the Reproducibility Divide: Open Source Software's Role in Standardizing Healthcare AI](https://mlsanity.com/paper/2603.03367/) (2026)
- [Bit-Identical Medical Deep Learning via Structured Orthogonal Initialization](https://mlsanity.com/paper/2603.28040/) (2026)
- [Croissant Tasks: A Metadata Format for Reproducible Machine Learning Evaluations](https://mlsanity.com/paper/2605.29786/) (2026)
- [RecourseBench: A Modular Framework for Reproducible Algorithmic Recourse Evaluation](https://mlsanity.com/paper/2606.16113/) (2026)

## Abstract (arXiv)

In the rapidly evolving fields of Artificial Intelligence (AI) and Machine Learning (ML), the reproducibility crisis underscores the urgent need for clear validation methodologies to maintain scientific integrity and encourage advancement. The crisis is compounded by the prevalent confusion over validation terminology. In response to this challenge, we introduce a framework that clarifies the roles and definitions of key validation efforts: repeatability, dependent and independent reproducibility, and direct and conceptual replicability. This structured framework aims to provide AI/ML researchers with the necessary clarity on these essential concepts, facilitating the appropriate design, conduct, and interpretation of validation studies. By articulating the nuances and specific roles of each type of validation study, we aim to enhance the reliability and trustworthiness of research findings and support the community's efforts to address reproducibility challenges effectively.