mlsanityData to 7 Oct 2026

Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)

arXiv 2003.12206 ↗First posted 27 Mar 2020cs.LG stat.MLcited by 318

Not on the map. Why?

Cited by 20 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

One of the challenges in machine learning research is to ensure that presented and published results are sound and reliable. Reproducibility, that is obtaining similar results as presented in a paper or talk, using the same code and data (when available), is a necessary step to verify the reliability of research findings. Reproducibility is also an important step to promote open and accessible research, thereby allowing the scientific community to quickly integrate new findings and convert ideas to practice. Reproducibility also promotes the use of robust experimental workflows, which potentially reduce unintentional errors. In 2019, the Neural Information Processing Systems (NeurIPS) conference, the premier international conference for research in machine learning, introduced a reproducibility program, designed to improve the standards across the community for how we conduct, communicate, and evaluate machine learning research. The program contained three components: a code submission policy, a community-wide reproducibility challenge, and the inclusion of the Machine Learning Reproducibility checklist as part of the paper submission process. In this paper, we describe each of these components, how it was deployed, as well as what we were able to learn from this initiative.

Key facts. Cited by 318 papers in the map (data through 7 Oct 2026). First posted on arXiv 27 Mar 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 28; 2022: 41; 2023: 38; 2024: 35; 2025: 46; 2026: 130.

318 papers.

Sort:

By area:

  1. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  2. 2021Trustworthy AI: From Principles to Practicescited by 206
  3. 2022BlenderBot 3: a deployed conversational agent that continually learns to responsibly engagecited by 197
  4. 2021Avalanche: an End-to-End Library for Continual Learningcited by 124
  5. 2024CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmarkcited by 115
  6. 2022The Fallacy of AI Functionalitycited by 88
  7. 2025Paper2Code: Automating Code Generation from Scientific Papers in Machine Learningcited by 67
  8. 2022Leakage and the Reproducibility Crisis in ML-based Sciencecited by 63
  9. 2021Synthetic Benchmarks for Scientific Research in Explainable Machine Learningcited by 59
  10. 2021Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarkingcited by 53
  11. 2023An Empirical Study of Pre-Trained Model Reuse in the Hugging Face Deep Learning Model Registrycited by 45
  12. 2022Reproducibility in Learningcited by 44
38 more
  1. 2022Open High-Resolution Satellite Imagery: The WorldStrat Dataset -- With Application to Super-Resolutioncited by 40
  2. 2021Bag of Tricks for Training Deeper Graph Neural Networks: A Comprehensive Benchmark Studycited by 36
  3. 2021How to avoid machine learning pitfalls: a guide for academic researcherscited by 34
  4. 2024Reproducibility in Machine Learning-based Research: Overview, Barriers and Driverscited by 34
  5. 2022Data Governance in the Age of Large-Scale Data-Driven Language Technologycited by 32
  6. 2021The Multi-Agent Behavior Dataset: Mouse Dyadic Social Interactionscited by 31
  7. 2022A Survey on Graph Counterfactual Explanations: Definitions, Methods, Evaluation, and Research Challengescited by 30
  8. 2023OCAtari: Object-Centric Atari 2600 Reinforcement Learning Environmentscited by 30
  9. 2021CLEVA-Compass: A Continual Learning EValuation Assessment Compass to Promote Research Transparency and Comparabilitycited by 28
  10. 2022Sources of Irreproducibility in Machine Learning: A Reviewcited by 28
  11. 2022Benchopt: Reproducible, efficient and collaborative optimization benchmarkscited by 27
  12. 2021Choice modelling in the age of machine learning -- discussion papercited by 26
  13. 2021Reusable Templates and Guides For Documenting Datasets and Models for Natural Language Processing and Generation: A Case Study of the HuggingFace and GEM Data and Model Cardscited by 24
  14. 2022Towards Training Reproducible Deep Learning Modelscited by 23
  15. 2022Replicable Banditscited by 22
  16. 2022Reproducibility in Optimization: Theoretical Framework and Limitscited by 21
  17. 2022Evaluate & Evaluation on the Hub: Better Best Practices for Data and Model Measurementscited by 21
  18. 2023Replicable Reinforcement Learningcited by 21
  19. 2023Replicability in Reinforcement Learningcited by 21
  20. 2024The Model Openness Framework: Promoting Completeness and Openness for Reproducibility, Transparency, and Usability in Artificial Intelligencecited by 21
  21. 2024Beyond Prompts: Dynamic Conversational Benchmarking of Large Language Modelscited by 20
  22. 2025LEMUR Neural Network Dataset: Towards Seamless AutoMLcited by 20
  23. 2022DaisyRec 2.0: Benchmarking Recommendation for Rigorous Evaluationcited by 19
  24. 2023Replicable Clusteringcited by 19
  25. 2024Position: Why We Must Rethink Empirical Research in Machine Learningcited by 19
  26. 2023Reproducibility in NLP: What Have We Learned from the Checklist?cited by 18
  27. 2022The worst of both worlds: A comparative analysis of errors in learning from data in psychology and machine learningcited by 17
  28. 2023Statistical Indistinguishability of Learning Algorithmscited by 17
  29. 2024Open RL Benchmark: Comprehensive Tracked Experiments for Reinforcement Learningcited by 17
  30. 2026MolmoWeb: Open Visual Web Agent and Open Data for the Open Webcited by 17
  31. 2024SzCORE: A Seizure Community Open-source Research Evaluation framework for the validation of EEG-based automated seizure detection algorithmscited by 16
  32. 2022SoK: The Impact of Unlabelled Data in Cyberthreat Detectioncited by 15
  33. 2023Reproducibility in Machine Learning-Driven Researchcited by 15
  34. 2024Public Domain 12M: A Highly Aesthetic Image-Text Dataset with Novel Governance Mechanismscited by 15
  35. 2023When Good and Reproducible Results are a Giant with Feet of Clay: The Importance of Software Quality in NLPcited by 14
  36. 2024Verifiable evaluations of machine learning models using zkSNARKscited by 14
  37. 2021Automated Evolutionary Approach for the Design of Composite Machine Learning Pipelinescited by 13
  38. 2021Automated Deep Learning: Neural Architecture Search Is Not the Endcited by 13

50 most cited of 318 All 318 (JSON)