# A Siren Song of Open Source Reproducibility

arXiv: https://arxiv.org/abs/2204.04372 · HTML: https://mlsanity.com/paper/2204.04372/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 61 references that are in the map.
- First posted on arXiv 9 Apr 2022.

## Builds on (most cited first, 26 of 26)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [Distributed Representations of Sentences and Documents](https://mlsanity.com/paper/1405.4053/) (2014)
- [TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems](https://mlsanity.com/paper/1603.04467/) (2016)
- [Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks](https://mlsanity.com/paper/2103.14749/) (2021)
- [Accounting for Variance in Machine Learning Benchmarks](https://mlsanity.com/paper/2103.03098/) (2021)
- [Do CIFAR-10 Classifiers Generalize to CIFAR-10?](https://mlsanity.com/paper/1806.00451/) (2018)
- [Should we really use post-hoc tests based on mean-ranks?](https://mlsanity.com/paper/1505.02288/) (2015)
- [word2vec Explained: deriving Mikolov et al.'s negative-sampling word-embedding method](https://mlsanity.com/paper/1402.3722/) (2014)
- [Troubling Trends in Machine Learning Scholarship](https://mlsanity.com/paper/1807.03341/) (2018)
- [A Step Toward Quantifying Independently Reproducible Machine Learning Research](https://mlsanity.com/paper/1909.06674/) (2019)
- [Do We Train on Test Data? Purging CIFAR of Near-Duplicates](https://mlsanity.com/paper/1902.00423/) (2019)
- [Randomness In Neural Network Training: Characterizing The Impact of Tooling](https://mlsanity.com/paper/2106.11872/) (2021)
- [Replicability Analysis for Natural Language Processing: Testing Significance with Multiple Datasets](https://mlsanity.com/paper/1709.09500/) (2017)
- [Identifying Statistical Bias in Dataset Replication](https://mlsanity.com/paper/2005.09619/) (2020)
- [An Open Review of OpenReview: A Critical Analysis of the Machine Learning Conference Review Process](https://mlsanity.com/paper/2010.05137/) (2020)
- ["Garbage In, Garbage Out" Revisited: What Do Machine Learning Application Papers Report About Human-Labeled Training Data?](https://mlsanity.com/paper/2107.02278/) (2021)
- [Reproducibility in Optimization: Theoretical Framework and Limits](https://mlsanity.com/paper/2202.04598/) (2022)
- [Research Reproducibility as a Survival Analysis](https://mlsanity.com/paper/2012.09932/) (2020)
- [Does the Market of Citations Reward Reproducible Work?](https://mlsanity.com/paper/2204.03829/) (2022)
- [Enabling End-To-End Machine Learning Replicability: A Case Study in Educational Data Mining](https://mlsanity.com/paper/1806.05208/) (2018)
- [dagger: A Python Framework for Reproducible Machine Learning Experiment Orchestration](https://mlsanity.com/paper/2006.07484/) (2020)

## Built on by (most cited first, 8 of 8)

- [Position: Why We Must Rethink Empirical Research in Machine Learning](https://mlsanity.com/paper/2405.02200/) (2024)
- [A Coreset Learning Reality Check](https://mlsanity.com/paper/2301.06163/) (2023)
- [Lempel-Ziv Networks](https://mlsanity.com/paper/2211.13250/) (2022)
- [You Don't Need Robust Machine Learning to Manage Adversarial Attack Risks](https://mlsanity.com/paper/2306.09951/) (2023)
- [Exploring the Sharpened Cosine Similarity](https://mlsanity.com/paper/2307.13855/) (2023)
- [Bayesian Meta-Analyses Could Be More: A Case Study in Trial of Labor After a Cesarean-section Outcomes and Complications](https://mlsanity.com/paper/2601.10089/) (2026)
- [Simpler Methods Work Better for L1 Penalized Logistic Models and Large Datasets](https://mlsanity.com/paper/2609.23995/) (2026)
- [KNN Implementation Details Can Dramatically Change Performance: An Example from Cover Trees](https://mlsanity.com/paper/2610.16940/) (2026)

## Abstract (arXiv)

As reproducibility becomes a greater concern, conferences have largely converged to a strategy of asking reviewers to indicate whether code was attached to a submission. This is part of a larger trend of taking action based on assumed ideals, without studying if those actions will yield the desired outcome. Our argument is that this focus on code for replication is misguided if we want to improve the state of reproducible research. This focus can be harmful -- we should not force code to be submitted. There is a lack of evidence for effective actions taken by conferences to encourage and reward reproducibility. We argue that venues must take more action to advance reproducible machine learning research today.