# The Fundamental Principles of Reproducibility

arXiv: https://arxiv.org/abs/2011.10098 · HTML: https://mlsanity.com/paper/2011.10098/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Nov 2020.

## Built on by (most cited first, 17 of 17)

- [Weak baselines and reporting biases lead to overoptimism in machine learning for fluid-related partial differential equations](https://mlsanity.com/paper/2407.07218/) (2024)
- [Leakage and the Reproducibility Crisis in ML-based Science](https://mlsanity.com/paper/2207.07048/) (2022)
- [Reproducibility in Machine Learning-based Research: Overview, Barriers and Drivers](https://mlsanity.com/paper/2406.14325/) (2024)
- [Sources of Irreproducibility in Machine Learning: A Review](https://mlsanity.com/paper/2204.07610/) (2022)
- [Position: Why We Must Rethink Empirical Research in Machine Learning](https://mlsanity.com/paper/2405.02200/) (2024)
- [Reflections on the Reproducibility of Commercial LLM Performance in Empirical Software Engineering Studies](https://mlsanity.com/paper/2510.25506/) (2025)
- [What is Reproducibility in Artificial Intelligence and Machine Learning Research?](https://mlsanity.com/paper/2407.10239/) (2024)
- [Benchmark Data Repositories for Better Benchmarking](https://mlsanity.com/paper/2410.24100/) (2024)
- [Reproducibility of Machine Learning: Terminology, Recommendations and Open Issues](https://mlsanity.com/paper/2302.12691/) (2023)
- [GENIUS: An Agentic AI Framework for Autonomous Design and Execution of Simulation Protocols](https://mlsanity.com/paper/2512.06404/) (2025)
- [Examining the Effect of Implementation Factors on Deep Learning Reproducibility](https://mlsanity.com/paper/2312.06633/) (2023)
- [Modeling the Economic Impacts of AI Openness Regulation](https://mlsanity.com/paper/2507.14193/) (2025)
- [From Model Performance to Claim: How a Change of Focus in Machine Learning Replicability Can Help Bridge the Responsibility Gap](https://mlsanity.com/paper/2404.13131/) (2024)
- [Reproducibility and Geometric Intrinsic Dimensionality: An Investigation on Graph Neural Network Research](https://mlsanity.com/paper/2403.08438/) (2024)
- [From Data to Decision: Data-Centric Infrastructure for Reproducible ML in Collaborative eScience](https://mlsanity.com/paper/2506.16051/) (2025)
- [A K-Means, Ward and DBSCAN repeatability study](https://mlsanity.com/paper/2512.19772/) (2025)
- [RECLAIM: Can Agents Reproduce the Claims of Machine Learning Papers?](https://mlsanity.com/paper/2609.28850/) (2026)

## Abstract (arXiv)

Reproducibility is a confused terminology. In this paper, I take a fundamental view on reproducibility rooted in the scientific method. The scientific method is analysed and characterised in order to develop the terminology required to define reproducibility. Further, the literature on reproducibility and replication is surveyed, and experiments are modeled as tasks and problem solving methods. Machine learning is used to exemplify the described approach. Based on the analysis, reproducibility is defined and three different types of reproducibility as well as four degrees of reproducibility are specified.