# Can citations tell us about a paper's reproducibility? A case study of machine learning papers

arXiv: https://arxiv.org/abs/2405.03977 · HTML: https://mlsanity.com/paper/2405.03977/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 27 references that are in the map.
- First posted on arXiv 7 May 2024.

## Builds on (most cited first, 10 of 10)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Structural Scaffolds for Citation Intent Classification in Scientific Publications](https://mlsanity.com/paper/1904.01608/) (2019)
- [The Semantic Scholar Open Data Platform](https://mlsanity.com/paper/2301.10140/) (2023)
- [A Step Toward Quantifying Independently Reproducible Machine Learning Research](https://mlsanity.com/paper/1909.06674/) (2019)
- [A Systematic Review of Reproducibility Research in Natural Language Processing](https://mlsanity.com/paper/2103.07929/) (2021)
- [Predicting the Reproducibility of Social and Behavioral Science Papers Using Supervised Learning Models](https://mlsanity.com/paper/2104.04580/) (2021)
- [A Study on Reproducibility and Replicability of Table Structure Recognition Methods](https://mlsanity.com/paper/2304.10439/) (2023)
- [Inline Citation Classification using Peripheral Context and Time-evolving Augmentation](https://mlsanity.com/paper/2303.00344/) (2023)
- [Laying foundations to quantify the "Effort of Reproducibility"](https://mlsanity.com/paper/2308.12580/) (2023)

## Built on by (most cited first, 2 of 2)

- [[Re] Network Deconvolution](https://mlsanity.com/paper/2410.01189/) (2024)
- [CC30k: A Citation Contexts Dataset for Reproducibility-Oriented Sentiment Analysis](https://mlsanity.com/paper/2511.07790/) (2025)

## Abstract (arXiv)

The iterative character of work in machine learning (ML) and artificial intelligence (AI) and reliance on comparisons against benchmark datasets emphasize the importance of reproducibility in that literature. Yet, resource constraints and inadequate documentation can make running replications particularly challenging. Our work explores the potential of using downstream citation contexts as a signal of reproducibility. We introduce a sentiment analysis framework applied to citation contexts from papers involved in Machine Learning Reproducibility Challenges in order to interpret the positive or negative outcomes of reproduction attempts. Our contributions include training classifiers for reproducibility-related contexts and sentiment analysis, and exploring correlations between citation context sentiment and reproducibility scores. Study data, software, and an artifact appendix are publicly available at https://github.com/lamps-lab/ccair-ai-reproducibility .