mlsanityData to 13 Oct 2026

A Siren Song of Open Source Reproducibility

arXiv 2204.04372 ↗First posted 9 Apr 2022cs.LG cs.AI cs.SEcited by 7

Not on the map. Why?

Abstract (arXiv)

As reproducibility becomes a greater concern, conferences have largely converged to a strategy of asking reviewers to indicate whether code was attached to a submission. This is part of a larger trend of taking action based on assumed ideals, without studying if those actions will yield the desired outcome. Our argument is that this focus on code for replication is misguided if we want to improve the state of reproducible research. This focus can be harmful -- we should not force code to be submitted. There is a lack of evidence for effective actions taken by conferences to encourage and reward reproducibility. We argue that venues must take more action to advance reproducible machine learning research today.

Key facts. Cited by 7 papers in the map (data through 13 Oct 2026). Builds on 26 of its 61 references that are in the map. First posted on arXiv 9 Apr 2022.

Builds on

26 of 61 references.

References · 26

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,093
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,223
  3. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,360
  4. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,510
  5. 2018Datasheets for Datasetscited by 1,377
  6. 2018Model Cards for Model Reportingcited by 1,076
  7. 2014Distributed Representations of Sentences and Documentscited by 677
  8. 2016TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systemscited by 662
  9. 2021Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarkscited by 391
  10. 2018Do CIFAR-10 Classifiers Generalize to CIFAR-10?cited by 198
  11. 2021Accounting for Variance in Machine Learning Benchmarkscited by 195
  12. 2015Should we really use post-hoc tests based on mean-ranks?cited by 116
14 more
  1. 2014word2vec Explained: deriving Mikolov et al.'s negative-sampling word-embedding methodcited by 112
  2. 2018Troubling Trends in Machine Learning Scholarshipcited by 108
  3. 2019A Step Toward Quantifying Independently Reproducible Machine Learning Researchcited by 77
  4. 2019Do We Train on Test Data? Purging CIFAR of Near-Duplicatescited by 61
  5. 2021Randomness In Neural Network Training: Characterizing The Impact of Toolingcited by 58
  6. 2017Replicability Analysis for Natural Language Processing: Testing Significance with Multiple Datasetscited by 32
  7. 2020Identifying Statistical Bias in Dataset Replicationcited by 24
  8. 2020An Open Review of OpenReview: A Critical Analysis of the Machine Learning Conference Review Processcited by 24
  9. 2021"Garbage In, Garbage Out" Revisited: What Do Machine Learning Application Papers Report About Human-Labeled Training Data?cited by 24
  10. 2022Reproducibility in Optimization: Theoretical Framework and Limitscited by 21
  11. 2020Research Reproducibility as a Survival Analysiscited by 14
  12. 2022Does the Market of Citations Reward Reproducible Work?cited by 7
  13. 2018Enabling End-To-End Machine Learning Replicability: A Case Study in Educational Data Miningcited by 4
  14. 2020dagger: A Python Framework for Reproducible Machine Learning Experiment Orchestrationcited by 4

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2022: 1; 2023: 3; 2024: 1; 2025: 0; 2026: 2.

7 papers.

  1. 2024Position: Why We Must Rethink Empirical Research in Machine Learningcited by 19
  2. 2023A Coreset Learning Reality Checkcited by 4
  3. 2022Lempel-Ziv Networkscited by 3
  4. 2023You Don't Need Robust Machine Learning to Manage Adversarial Attack Riskscited by 3
  5. 2023Exploring the Sharpened Cosine Similaritycited by 1
  6. 2026Bayesian Meta-Analyses Could Be More: A Case Study in Trial of Labor After a Cesarean-section Outcomes and Complicationscited by 0
  7. 2026Simpler Methods Work Better for L1 Penalized Logistic Models and Large Datasetscited by 0