mlsanityData to 7 Oct 2026

A Step Toward Quantifying Independently Reproducible Machine Learning Research

arXiv 1909.06674 ↗First posted 14 Sep 2019cs.LG cs.AI cs.DLcited by 77

Not on the map. Why?

Abstract (arXiv)

What makes a paper independently reproducible? Debates on reproducibility center around intuition or assumptions but lack empirical results. Our field focuses on releasing code, which is important, but is not sufficient for determining reproducibility. We take the first step toward a quantifiable answer by manually attempting to implement 255 papers published from 1984 until 2017, recording features of each paper, and performing statistical analysis of the results. For each paper, we did not look at the authors code, if released, in order to prevent bias toward discrepancies between code and paper.

Key facts. Cited by 77 papers in the map (data through 7 Oct 2026). First posted on arXiv 14 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 10; 2022: 16; 2023: 13; 2024: 14; 2025: 9; 2026: 15.

77 papers.

Sort:

By area:

  1. 2021Accounting for Variance in Machine Learning Benchmarkscited by 195
  2. 2021Avalanche: an End-to-End Library for Continual Learningcited by 124
  3. 2024CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmarkcited by 115
  4. 2021Randomness In Neural Network Training: Characterizing The Impact of Toolingcited by 57
  5. 2021Just What do You Think You're Doing, Dave?' A Checklist for Responsible Data Use in NLPcited by 47
  6. 2022Accountability in an Algorithmic Society: Relationality, Responsibility, and Robustness in Machine Learningcited by 47
  7. 2024Reproducibility in Machine Learning-based Research: Overview, Barriers and Driverscited by 34
  8. 2022Sources of Irreproducibility in Machine Learning: A Reviewcited by 28
  9. 2022Benchopt: Reproducible, efficient and collaborative optimization benchmarkscited by 27
  10. 2022Towards Training Reproducible Deep Learning Modelscited by 23
  11. 2022Evaluate & Evaluation on the Hub: Better Best Practices for Data and Model Measurementscited by 21
  12. 2021Hyperparameter Optimization Is Deceiving Us, and How to Stop Itcited by 20
38 more
  1. 2022DaisyRec 2.0: Benchmarking Recommendation for Rigorous Evaluationcited by 19
  2. 2024Position: Why We Must Rethink Empirical Research in Machine Learningcited by 19
  3. 2021A Framework for Cluster and Classifier Evaluation in the Absence of Reference Labelscited by 16
  4. 2023When Good and Reproducible Results are a Giant with Feet of Clay: The Importance of Software Quality in NLPcited by 14
  5. 2023REFORMS: Reporting Standards for Machine Learning Based Sciencecited by 13
  6. 2026stable-worldmodel: A Platform for Reproducible World Modeling Research and Evaluationcited by 13
  7. 2024Assemblage: Automatic Binary Dataset Construction for Machine Learningcited by 12
  8. 2022Metaethical Perspectives on 'Benchmarking' AI Ethicscited by 11
  9. 2022Modeling the Machine Learning Multiversecited by 11
  10. 2023Reproducibility in Multiple Instance Learning: A Case For Algorithmic Unit Testscited by 11
  11. 2024What is Reproducibility in Artificial Intelligence and Machine Learning Research?cited by 11
  12. 2022A Revealing Large-Scale Evaluation of Unsupervised Anomaly Detection Algorithmscited by 10
  13. 2024Benchmark Data Repositories for Better Benchmarkingcited by 10
  14. 2022Automatic Analysis of Available Source Code of Top Artificial Intelligence Conference Paperscited by 9
  15. 2026SciCoQA: Quality Assurance for Scientific Paper--Code Alignmentcited by 9
  16. 2023Reproducibility of Machine Learning: Terminology, Recommendations and Open Issuescited by 8
  17. 2024Neural Circuit Diagrams: Robust Diagrams for the Communication, Implementation, and Analysis of Deep Learning Architecturescited by 8
  18. 2024What Do Machine Learning Researchers Mean by "Reproducible"?cited by 8
  19. 2022Does the Market of Citations Reward Reproducible Work?cited by 7
  20. 2022A Siren Song of Open Source Reproducibilitycited by 7
  21. 2022Prescriptive and Descriptive Approaches to Machine-Learning Transparencycited by 7
  22. 2022Non-Determinism and the Lawlessness of Machine Learning Codecited by 7
  23. 2022Reproducibility of the Methods in Medical Imaging with Deep Learningcited by 5
  24. 2025AI-Generated Code Is Not Reproducible (Yet): An Empirical Study of Dependency Gaps in LLM-Based Coding Agentscited by 5
  25. 2021Meta-repository of screening mammography classifierscited by 4
  26. 2022When Bioprocess Engineering Meets Machine Learning: A Survey from the Perspective of Automated Bioprocess Developmentcited by 4
  27. 2023A Coreset Learning Reality Checkcited by 4
  28. 2023The NCI Imaging Data Commons as a platform for reproducible research in computational pathologycited by 4
  29. 2023A Study on Reproducibility and Replicability of Table Structure Recognition Methodscited by 4
  30. 2021On the reproducibility of fully convolutional neural networks for modeling time-space evolving physical systemscited by 3
  31. 2021Challenges for cognitive decoding using deep learning methodscited by 3
  32. 2023You Don't Need Robust Machine Learning to Manage Adversarial Attack Riskscited by 3
  33. 2023Examining the Effect of Implementation Factors on Deep Learning Reproducibilitycited by 3
  34. 2024Questionable practices in machine learningcited by 3
  35. 2026Benchmark of Benchmarks: Unpacking Influence and Code Repository Quality in LLM Safety Benchmarkscited by 3
  36. 2024Can citations tell us about a paper's reproducibility? A case study of machine learning paperscited by 2
  37. 2026FactReview: Evidence-Grounded Peer Review with Execution-Based Claim Verificationcited by 2
  38. 2026Preregistration for Experiments with AI Agentscited by 2

50 most cited of 77 All 77 (JSON)