mlsanityData to 7 Oct 2026

A Systematic Review of Reproducibility Research in Natural Language Processing

arXiv 2103.07929 ↗First posted 14 Mar 2021cs.CLcited by 34

Not on the map. Why?

Abstract (arXiv)

Against the background of what has been termed a reproducibility crisis in science, the NLP field is becoming increasingly interested in, and conscientious about, the reproducibility of its results. The past few years have seen an impressive range of new initiatives, events and active research in the area. However, the field is far from reaching a consensus about how reproducibility should be defined, measured and addressed, with diversity of views currently increasing rather than converging. With this focused contribution, we aim to provide a wide-angle, and as near as possible complete, snapshot of current work on reproducibility in NLP, delineating differences and similarities, and providing pointers to common denominators.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Builds on 16 of its 83 references that are in the map. First posted on arXiv 14 Mar 2021.

Builds on

16 of 83 references.

References · 16

  1. 2017Deep Learning for Hate Speech Detection in Tweetscited by 180
  2. 2018A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddingscited by 145
  3. 2016Harnessing Deep Neural Networks with Logic Rulescited by 135
  4. 2015Effective LSTMs for Target-Dependent Sentiment Classificationcited by 82
  5. 2017Neural Language Modeling by Jointly Learning Syntax and Lexiconcited by 57
  6. 2014Learning to Win by Reading Manuals in a Monte-Carlo Frameworkcited by 39
  7. 2016Multilingual Part-of-Speech Tagging with Bidirectional Long Short-Term Memory Models and Auxiliary Losscited by 30
  8. 2018Morphosyntactic Tagging with a Meta-BiLSTM Model over Context Sensitive Token Encodingscited by 18
  9. 2018Experiments with Universal CEFR Classificationcited by 17
  10. 2017Sustainable computational science: the ReScience initiativecited by 10
  11. 2018Bringing replication and reproduction together with generalisability in NLP: Three reproduction studies for Target Dependent Sentiment Analysiscited by 4
  12. 2019A Robust Self-Learning Method for Fully Unsupervised Cross-Lingual Mappings of Word Embeddings: Making the Method Robustly Reproducible as Wellcited by 4
4 more
  1. 2018ETH-DS3Lab at SemEval-2018 Task 7: Effectively Combining Recurrent and Convolutional Neural Networks for Relation Classification and Extractioncited by 2
  2. 2016Learning Semantically and Additively Compositional Distributional Representationscited by 1
  3. 2018NLITrans at SemEval-2018 Task 12: Transfer of Semantic Knowledge for Argument Comprehensioncited by 1
  4. 2018SNU_IDS at SemEval-2018 Task 12: Sentence Encoder with Contextualized Vectors for Argument Reasoning Comprehensioncited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 6; 2023: 11; 2024: 6; 2025: 3; 2026: 3.

34 papers.

Sort:

By area:

  1. 2024CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmarkcited by 115
  2. 2023Efficient Benchmarking of Language Modelscited by 71
  3. 2022GEMv2: Multilingual NLG Benchmarking in a Single Line of Codecited by 36
  4. 2024Reproducibility in Machine Learning-based Research: Overview, Barriers and Driverscited by 34
  5. 2022Sources of Irreproducibility in Machine Learning: A Reviewcited by 28
  6. 2023Position: Key Claims in LLM Research Have a Long Tail of Footnotescited by 20
  7. 2021A Review of Human Evaluation for Style Transfercited by 19
  8. 2022Vote'n'Rank: Revision of Benchmarking with Social Choice Theorycited by 15
  9. 2023Reproducibility in Machine Learning-Driven Researchcited by 15
  10. 2024Public Domain 12M: A Highly Aesthetic Image-Text Dataset with Novel Governance Mechanismscited by 15
  11. 2023When Good and Reproducible Results are a Giant with Feet of Clay: The Importance of Software Quality in NLPcited by 14
  12. 2023iLab at SemEval-2023 Task 11 Le-Wi-Di: Modelling Disagreement or Modelling Perspectives?cited by 10
22 more
  1. 2021How May I Help You? Using Neural Text Simplification to Improve Downstream NLP Taskscited by 9
  2. 2021Is Stance Detection Topic-Independent and Cross-topic Generalizable? -- A Reproduction Studycited by 9
  3. 2022Reproducibility Issues for BERT-based Evaluation Metricscited by 9
  4. 2022Experimental Standards for Deep Learning in Natural Language Processing Researchcited by 9
  5. 2022Quantified Reproducibility Assessment of NLP Resultscited by 8
  6. 2023A Two-Sided Discussion of Preregistration of NLP Researchcited by 6
  7. 2023Towards Inferential Reproducibility of Machine Learning Researchcited by 4
  8. 2023NLP Reproducibility For All: Understanding Experiences of Beginnerscited by 4
  9. 2024Can citations tell us about a paper's reproducibility? A case study of machine learning paperscited by 2
  10. 2021Quantifying Reproducibility in NLP and MLcited by 1
  11. 2025Introducing HALC: A general pipeline for the systematic and reliable construction of prompts for automated coding with LLMs in the computational social sciencescited by 1
  12. 2021Small data problems in political research: a critical replication studycited by 0
  13. 2023Investigating Reproducibility at Interspeech Conferences: A Longitudinal and Comparative Perspectivecited by 0
  14. 2023Cyber Security Requirements for Platforms Enhancing AI Reproducibilitycited by 0
  15. 2023An End-to-End System for Reproducibility Assessment of Source Code Repositories via Their Readmescited by 0
  16. 2024A Second Look on BASS -- Boosting Abstractive Summarization with Unified Semantic Graphs -- A Replication Studycited by 0
  17. 2024Reproducing the Metric-Based Evaluation of a Set of Controllable Text Generation Techniquescited by 0
  18. 2025Reducing Variability of Multiple Instance Learning Methods for Digital Pathologycited by 0
  19. 2025Standardising the NLP Workflow: A Framework for Reproducible Linguistic Analysiscited by 0
  20. 2026The Costs of Reproducibility in Music Separation Research: a Replication of Band-Split RNNcited by 0
  21. 2026Croissant Tasks: A Metadata Format for Reproducible Machine Learning Evaluationscited by 0
  22. 2026Papers Without Code: Availability of GitHub Repositories Linked in *CL Publicationscited by 0