# Does the Market of Citations Reward Reproducible Work?

arXiv: https://arxiv.org/abs/2204.03829 · HTML: https://mlsanity.com/paper/2204.03829/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 47 references that are in the map.
- First posted on arXiv 8 Apr 2022.

## Builds on (most cited first, 11 of 11)

- [Accounting for Variance in Machine Learning Benchmarks](https://mlsanity.com/paper/2103.03098/) (2021)
- [Composable Effects for Flexible and Accelerated Probabilistic Programming in NumPyro](https://mlsanity.com/paper/1912.11554/) (2019)
- [A Step Toward Quantifying Independently Reproducible Machine Learning Research](https://mlsanity.com/paper/1909.06674/) (2019)
- [Growth rates of modern science: A bibliometric analysis based on the number of publications and cited references](https://mlsanity.com/paper/1402.4578/) (2014)
- [Replicability Analysis for Natural Language Processing: Testing Significance with Multiple Datasets](https://mlsanity.com/paper/1709.09500/) (2017)
- [Research Reproducibility as a Survival Analysis](https://mlsanity.com/paper/2012.09932/) (2020)
- [Predicting the long-term citation impact of recent publications](https://mlsanity.com/paper/1503.09156/) (2015)
- [Enabling End-To-End Machine Learning Replicability: A Case Study in Educational Data Mining](https://mlsanity.com/paper/1806.05208/) (2018)
- [Regression for citation data: An evaluation of different methods](https://mlsanity.com/paper/1510.08877/) (2015)
- [Inferring the causal effect of journals on citations](https://mlsanity.com/paper/1912.08648/) (2019)
- [Shorter Distances between Papers over Time are Due to More Cross-Field References and Increased Citation Rate to Higher Impact Papers](https://mlsanity.com/paper/1911.04548/) (2019)

## Built on by (most cited first, 8 of 8)

- [Experimental Standards for Deep Learning in Natural Language Processing Research](https://mlsanity.com/paper/2204.06251/) (2022)
- [A Siren Song of Open Source Reproducibility](https://mlsanity.com/paper/2204.04372/) (2022)
- [A Coreset Learning Reality Check](https://mlsanity.com/paper/2301.06163/) (2023)
- [More Rigorous Software Engineering Would Improve Reproducibility in Machine Learning Research](https://mlsanity.com/paper/2502.00902/) (2025)
- [Generalizability of experimental studies](https://mlsanity.com/paper/2406.17374/) (2024)
- [AI Research is not Magic, it has to be Reproducible and Responsible: Challenges in the AI field from the Perspective of its PhD Students](https://mlsanity.com/paper/2408.06847/) (2024)
- [Simpler Methods Work Better for L1 Penalized Logistic Models and Large Datasets](https://mlsanity.com/paper/2609.23995/) (2026)
- [Kernel-based External Validity of Experimental Studies](https://mlsanity.com/paper/2610.13625/) (2026)

## Abstract (arXiv)

The field of bibliometrics, studying citations and behavior, is critical to the discussion of reproducibility. Citations are one of the primary incentive and reward systems for academic work, and so we desire to know if this incentive rewards reproducible work. Yet to the best of our knowledge, only one work has attempted to look at this combined space, concluding that non-reproducible work is more highly cited. We show that answering this question is more challenging than first proposed, and subtle issues can inhibit a robust conclusion. To make inferences with more robust behavior, we propose a hierarchical Bayesian model that incorporates the citation rate over time, rather than the total number of citations after a fixed amount of time. In doing so we show that, under current evidence the answer is more likely that certain fields of study such as Medicine and Machine Learning (ML) do correlate reproducible works with more citations, but other fields appear to have no relationship. Further, we find that making code available and thoroughly referencing prior works appear to also positively correlate with increased citations. Our code and data can be found at https://github.com/EdwardRaff/ReproducibleCitations .