# A Systematic Review of Reproducibility Research in Natural Language Processing

arXiv: https://arxiv.org/abs/2103.07929 · HTML: https://mlsanity.com/paper/2103.07929/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 83 references that are in the map.
- First posted on arXiv 14 Mar 2021.

## Builds on (most cited first, 16 of 16)

- [Deep Learning for Hate Speech Detection in Tweets](https://mlsanity.com/paper/1706.00188/) (2017)
- [A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings](https://mlsanity.com/paper/1805.06297/) (2018)
- [Harnessing Deep Neural Networks with Logic Rules](https://mlsanity.com/paper/1603.06318/) (2016)
- [Effective LSTMs for Target-Dependent Sentiment Classification](https://mlsanity.com/paper/1512.01100/) (2015)
- [Neural Language Modeling by Jointly Learning Syntax and Lexicon](https://mlsanity.com/paper/1711.02013/) (2017)
- [Learning to Win by Reading Manuals in a Monte-Carlo Framework](https://mlsanity.com/paper/1401.5390/) (2014)
- [Multilingual Part-of-Speech Tagging with Bidirectional Long Short-Term Memory Models and Auxiliary Loss](https://mlsanity.com/paper/1604.05529/) (2016)
- [Morphosyntactic Tagging with a Meta-BiLSTM Model over Context Sensitive Token Encodings](https://mlsanity.com/paper/1805.08237/) (2018)
- [Experiments with Universal CEFR Classification](https://mlsanity.com/paper/1804.06636/) (2018)
- [Sustainable computational science: the ReScience initiative](https://mlsanity.com/paper/1707.04393/) (2017)
- [Bringing replication and reproduction together with generalisability in NLP: Three reproduction studies for Target Dependent Sentiment Analysis](https://mlsanity.com/paper/1806.05219/) (2018)
- [A Robust Self-Learning Method for Fully Unsupervised Cross-Lingual Mappings of Word Embeddings: Making the Method Robustly Reproducible as Well](https://mlsanity.com/paper/1912.01706/) (2019)
- [ETH-DS3Lab at SemEval-2018 Task 7: Effectively Combining Recurrent and Convolutional Neural Networks for Relation Classification and Extraction](https://mlsanity.com/paper/1804.02042/) (2018)
- [Learning Semantically and Additively Compositional Distributional Representations](https://mlsanity.com/paper/1606.02461/) (2016)
- [NLITrans at SemEval-2018 Task 12: Transfer of Semantic Knowledge for Argument Comprehension](https://mlsanity.com/paper/1804.08266/) (2018)
- [SNU_IDS at SemEval-2018 Task 12: Sentence Encoder with Contextualized Vectors for Argument Reasoning Comprehension](https://mlsanity.com/paper/1805.07049/) (2018)

## Built on by (most cited first, 30 of 34)

- [CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark](https://mlsanity.com/paper/2409.11363/) (2024)
- [Efficient Benchmarking of Language Models](https://mlsanity.com/paper/2308.11696/) (2023)
- [GEMv2: Multilingual NLG Benchmarking in a Single Line of Code](https://mlsanity.com/paper/2206.11249/) (2022)
- [Reproducibility in Machine Learning-based Research: Overview, Barriers and Drivers](https://mlsanity.com/paper/2406.14325/) (2024)
- [Sources of Irreproducibility in Machine Learning: A Review](https://mlsanity.com/paper/2204.07610/) (2022)
- [Position: Key Claims in LLM Research Have a Long Tail of Footnotes](https://mlsanity.com/paper/2308.07120/) (2023)
- [A Review of Human Evaluation for Style Transfer](https://mlsanity.com/paper/2106.04747/) (2021)
- [Vote'n'Rank: Revision of Benchmarking with Social Choice Theory](https://mlsanity.com/paper/2210.05769/) (2022)
- [Reproducibility in Machine Learning-Driven Research](https://mlsanity.com/paper/2307.10320/) (2023)
- [Public Domain 12M: A Highly Aesthetic Image-Text Dataset with Novel Governance Mechanisms](https://mlsanity.com/paper/2410.23144/) (2024)
- [When Good and Reproducible Results are a Giant with Feet of Clay: The Importance of Software Quality in NLP](https://mlsanity.com/paper/2303.16166/) (2023)
- [Experimental Standards for Deep Learning in Natural Language Processing Research](https://mlsanity.com/paper/2204.06251/) (2022)
- [iLab at SemEval-2023 Task 11 Le-Wi-Di: Modelling Disagreement or Modelling Perspectives?](https://mlsanity.com/paper/2305.06074/) (2023)
- [How May I Help You? Using Neural Text Simplification to Improve Downstream NLP Tasks](https://mlsanity.com/paper/2109.04604/) (2021)
- [Is Stance Detection Topic-Independent and Cross-topic Generalizable? -- A Reproduction Study](https://mlsanity.com/paper/2110.07693/) (2021)
- [Reproducibility Issues for BERT-based Evaluation Metrics](https://mlsanity.com/paper/2204.00004/) (2022)
- [Quantified Reproducibility Assessment of NLP Results](https://mlsanity.com/paper/2204.05961/) (2022)
- [A Two-Sided Discussion of Preregistration of NLP Research](https://mlsanity.com/paper/2302.10086/) (2023)
- [Towards Inferential Reproducibility of Machine Learning Research](https://mlsanity.com/paper/2302.04054/) (2023)
- [NLP Reproducibility For All: Understanding Experiences of Beginners](https://mlsanity.com/paper/2305.16579/) (2023)
- [Can citations tell us about a paper's reproducibility? A case study of machine learning papers](https://mlsanity.com/paper/2405.03977/) (2024)
- [Introducing HALC: A general pipeline for the systematic and reliable construction of prompts for automated coding with LLMs in the computational social sciences](https://mlsanity.com/paper/2507.21831/) (2025)
- [Quantifying Reproducibility in NLP and ML](https://mlsanity.com/paper/2109.01211/) (2021)
- [Small data problems in political research: a critical replication study](https://mlsanity.com/paper/2109.12911/) (2021)
- [Investigating Reproducibility at Interspeech Conferences: A Longitudinal and Comparative Perspective](https://mlsanity.com/paper/2306.10033/) (2023)
- [Cyber Security Requirements for Platforms Enhancing AI Reproducibility](https://mlsanity.com/paper/2309.15525/) (2023)
- [An End-to-End System for Reproducibility Assessment of Source Code Repositories via Their Readmes](https://mlsanity.com/paper/2310.09634/) (2023)
- [A Second Look on BASS -- Boosting Abstractive Summarization with Unified Semantic Graphs -- A Replication Study](https://mlsanity.com/paper/2403.02930/) (2024)
- [Reproducing the Metric-Based Evaluation of a Set of Controllable Text Generation Techniques](https://mlsanity.com/paper/2405.07875/) (2024)
- [Reducing Variability of Multiple Instance Learning Methods for Digital Pathology](https://mlsanity.com/paper/2507.00292/) (2025)

## Abstract (arXiv)

Against the background of what has been termed a reproducibility crisis in science, the NLP field is becoming increasingly interested in, and conscientious about, the reproducibility of its results. The past few years have seen an impressive range of new initiatives, events and active research in the area. However, the field is far from reaching a consensus about how reproducibility should be defined, measured and addressed, with diversity of views currently increasing rather than converging. With this focused contribution, we aim to provide a wide-angle, and as near as possible complete, snapshot of current work on reproducibility in NLP, delineating differences and similarities, and providing pointers to common denominators.