# CC30k: A Citation Contexts Dataset for Reproducibility-Oriented Sentiment Analysis

arXiv: https://arxiv.org/abs/2511.07790 · HTML: https://mlsanity.com/paper/2511.07790/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 63 references that are in the map.
- First posted on arXiv 11 Nov 2025.

## Builds on (most cited first, 21 of 21)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [Learning from Noisy Labels with Deep Neural Networks: A Survey](https://mlsanity.com/paper/2007.08199/) (2020)
- [SPECTER: Document-level Representation Learning using Citation-informed Transformers](https://mlsanity.com/paper/2004.07180/) (2020)
- [Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets](https://mlsanity.com/paper/1906.05474/) (2019)
- [Can language models learn from explanations in context?](https://mlsanity.com/paper/2204.02329/) (2022)
- [Structural Scaffolds for Citation Intent Classification in Scientific Publications](https://mlsanity.com/paper/1904.01608/) (2019)
- [The Semantic Scholar Open Data Platform](https://mlsanity.com/paper/2301.10140/) (2023)
- [A Step Toward Quantifying Independently Reproducible Machine Learning Research](https://mlsanity.com/paper/1909.06674/) (2019)
- [Verifiable Generation with Subsentence-Level Fine-Grained Citations](https://mlsanity.com/paper/2406.06125/) (2024)
- [Reproducibility, Replicability, and Repeatability: A survey of reproducible research with a focus on high performance computing](https://mlsanity.com/paper/2402.07530/) (2024)
- [Context-Enhanced Language Models for Generating Multi-Paper Citations](https://mlsanity.com/paper/2404.13865/) (2024)
- [A Study on Reproducibility and Replicability of Table Structure Recognition Methods](https://mlsanity.com/paper/2304.10439/) (2023)
- [Language Representation Models for Fine-Grained Sentiment Classification](https://mlsanity.com/paper/2005.13619/) (2020)
- [Laying foundations to quantify the "Effort of Reproducibility"](https://mlsanity.com/paper/2308.12580/) (2023)
- [Can citations tell us about a paper's reproducibility? A case study of machine learning papers](https://mlsanity.com/paper/2405.03977/) (2024)

## Abstract (arXiv)

Sentiments about the reproducibility of cited papers in downstream literature offer community perspectives and have shown as a promising signal of the actual reproducibility of published findings. To train effective models to effectively predict reproducibility-oriented sentiments and further systematically study their correlation with reproducibility, we introduce the CC30k dataset, comprising a total of 30,734 citation contexts in machine learning papers. Each citation context is labeled with one of three reproducibility-oriented sentiment labels: Positive, Negative, or Neutral, reflecting the cited paper's perceived reproducibility or replicability. Of these, 25,829 are labeled through crowdsourcing, supplemented with negatives generated through a controlled pipeline to counter the scarcity of negative labels. Unlike traditional sentiment analysis datasets, CC30k focuses on reproducibility-oriented sentiments, addressing a research gap in resources for computational reproducibility studies. The dataset was created through a pipeline that includes robust data cleansing, careful crowd selection, and thorough validation. The resulting dataset achieves a labeling accuracy of 94%. We then demonstrated that the performance of three large language models significantly improves on the reproducibility-oriented sentiment classification after fine-tuning using our dataset. The dataset lays the foundation for large-scale assessments of the reproducibility of machine learning papers. The CC30k dataset and the Jupyter notebooks used to produce and analyze the dataset are publicly available at https://github.com/lamps-lab/CC30k .