mlsanityData to 14 Oct 2026

What Do Machine Learning Researchers Mean by "Reproducible"?

arXiv 2412.03854 ↗First posted 5 Dec 2024cs.LG cs.AI stat.MLcited by 9

Not on the map. Why?

Abstract (arXiv)

The concern that Artificial Intelligence (AI) and Machine Learning (ML) are entering a "reproducibility crisis" has spurred significant research in the past few years. Yet with each paper, it is often unclear what someone means by "reproducibility". Our work attempts to clarify the scope of "reproducibility" as displayed by the community at large. In doing so, we propose to refine the research to eight general topic areas. In this light, we see that each of these areas contains many works that do not advertise themselves as being about "reproducibility", in part because they go back decades before the matter came to broader attention.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 38 of its 158 references that are in the map. First posted on arXiv 5 Dec 2024.

Builds on

38 of 158 references.

References · 38

  1. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  2. 2020What is the State of Neural Network Pruning?cited by 583
  3. 2020Are we done with ImageNet?cited by 333
  4. 2020A Metric Learning Reality Checkcited by 321
  5. 2016Data Programming: Creating Large Training Sets, Quicklycited by 246
  6. 2021Accounting for Variance in Machine Learning Benchmarkscited by 201
  7. 2022Why do tree-based models still outperform deep learning on tabular data?cited by 199
  8. 2019Show Your Work: Improved Reporting of Experimental Resultscited by 169
  9. 2021HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPOcited by 112
  10. 2021Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Researchcited by 102
  11. 2021Towards a Rigorous Evaluation of Time-series Anomaly Detectioncited by 97
  12. 2019A Step Toward Quantifying Independently Reproducible Machine Learning Researchcited by 77
26 more
  1. 2015The Ladder: A Reliable Leaderboard for Machine Learning Competitionscited by 69
  2. 2021Cross-validation: what does it estimate and how well does it do it?cited by 65
  3. 2019Do We Train on Test Data? Purging CIFAR of Near-Duplicatescited by 62
  4. 2021Randomness In Neural Network Training: Characterizing The Impact of Toolingcited by 59
  5. 2022Benchopt: Reproducible, efficient and collaborative optimization benchmarkscited by 27
  6. 2021Learning with Holographic Reduced Representationscited by 25
  7. 2022Replicable Banditscited by 22
  8. 2022Reproducibility in Optimization: Theoretical Framework and Limitscited by 21
  9. 2023Replicability in Reinforcement Learningcited by 21
  10. 2021Hyperparameter Optimization Is Deceiving Us, and How to Stop Itcited by 20
  11. 2023Statistical Indistinguishability of Learning Algorithmscited by 17
  12. 2020On the Replicability and Reproducibility of Deep Learning in Software Engineeringcited by 16
  13. 2020Research Reproducibility as a Survival Analysiscited by 16
  14. 2023When Good and Reproducible Results are a Giant with Feet of Clay: The Importance of Software Quality in NLPcited by 14
  15. 2024Assemblage: Automatic Binary Dataset Construction for Machine Learningcited by 12
  16. 2018Is rotation forest the best classifier for problems with continuous features?cited by 11
  17. 2021Leveraging Uncertainty for Improved Static Malware Detection Under Extreme False Positive Constraintscited by 11
  18. 2023Reproducibility in Multiple Instance Learning: A Case For Algorithmic Unit Testscited by 11
  19. 2024Is Function Similarity Over-Engineered? Building a Benchmarkcited by 6
  20. 2022Building a Culture of Reproducibility in Academic Researchcited by 5
  21. 2023A Coreset Learning Reality Checkcited by 5
  22. 2020dagger: A Python Framework for Reproducible Machine Learning Experiment Orchestrationcited by 4
  23. 2023Evaluating Model Performance in Medical Datasets Over Timecited by 4
  24. 2023NLP Reproducibility For All: Understanding Experiences of Beginnerscited by 4
  25. 2022Code Smells in Machine Learning Systemscited by 3
  26. 2023Optimal Guarantees for Algorithmic Reproducibility and Gradient Complexity in Convex Optimizationcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 7.

9 papers.

  1. 2025Building Trustworthy AI for Materials Discovery: From Autonomous Laboratories to Z-scorescited by 2
  2. 2026Jailbreak Foundry: From Papers to Runnable Attacks for Reproducible Benchmarkingcited by 2
  3. 2026Assessing Reproducibility in Evolutionary Computation: A Case Study using Human- and LLM-based Assessmentcited by 1
  4. 2025A Systematic Review on the Generative AI Applications in Human Medical Genomicscited by 0
  5. 2026Unrequited Emotions: Investigating the Gaps in Motivation and Practice in Speech Emotion Recognition Researchcited by 0
  6. 2026Simpler Methods Work Better for L1 Penalized Logistic Models and Large Datasetscited by 0
  7. 2026Training Witnesses: Trusting the Training without Trusting the Trainercited by 0
  8. 2026Position: Let's Strengthen Verifiability If We Can't Enforce Reproducibilitycited by 0
  9. 2026KNN Implementation Details Can Dramatically Change Performance: An Example from Cover Treescited by 0