mlsanityData to 7 Oct 2026

Missing Information, Unresponsive Authors, Experimental Flaws: The Impossibility of Assessing the Reproducibility of Previous Human Evaluations in NLP

arXiv 2305.01633 ↗First posted 2 May 2023cs.CLcited by 21

Not on the map. Why?

Abstract (arXiv)

We report our efforts in identifying a set of previous human evaluations in NLP that would be suitable for a coordinated study examining what makes human evaluations in NLP more/less reproducible. We present our results and findings, which include that just 13% of papers had (i) sufficiently low barriers to reproduction, and (ii) enough obtainable information, to be considered for reproduction, and that all but one of the experiments we selected for reproduction was discovered to have flaws that made the meaningfulness of conducting a reproduction questionable. As a result, we had to change our coordinated study design from a reproduce approach to a standardise-then-reproduce-twice approach. Our overall (negative) finding that the great majority of human evaluations in NLP is not repeatable and/or not reproducible and/or too flawed to justify reproduction, paints a dire picture, but presents an opportunity for a rethink about how to design and report human evaluations in NLP.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 2 of its 29 references that are in the map. First posted on arXiv 2 May 2023.

Builds on

2 of 29 references.

References · 2

  1. 2021The Human Evaluation Datasheet 1.0: A Template for Recording Details of Human Evaluation Experiments in NLPcited by 19
  2. 2022Quantified Reproducibility Assessment of NLP Resultscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 9; 2025: 1; 2026: 3.

21 papers.

Sort:

By area:

  1. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 314
  2. 2023KoLA: Carefully Benchmarking World Knowledge of Large Language Modelscited by 85
  3. 2023GPTEval: A Survey on Assessments of ChatGPT and GPT-4cited by 73
  4. 2024Are LLM-Judges Robust to Expressions of Uncertainty? Investigating the effect of Epistemic Markers on LLM-based Evaluationcited by 29
  5. 2024How Good Are LLMs for Literary Translation, Really? Literary Translation Evaluation with Humans and LLMscited by 23
  6. 2024Beyond Probabilities: Unveiling the Misalignment in Evaluating Large Language Modelscited by 19
  7. 2024Automatic Metrics in Natural Language Generation: A Survey of Current Evaluation Practicescited by 19
  8. 2023Understanding Counterspeech for Online Harm Mitigationcited by 17
  9. 2024Evaluating Diversity in Automatic Poetry Generationcited by 17
  10. 2024Verifiable evaluations of machine learning models using zkSNARKscited by 14
  11. 2024The Promises and Pitfalls of LLM Annotations in Dataset Labeling: a Case Study on Media Bias Detectioncited by 14
  12. 2023Evaluating Task-oriented Dialogue Systems: A Systematic Review of Measures, Constructs and their Operationalisationscited by 9
9 more
  1. 2025SPHERE: An Evaluation Card for Human-AI Systemscited by 9
  2. 2023Cross-lingual Cross-temporal Summarization: Dataset, Models, Evaluationcited by 8
  3. 2023Evaluating AI systems under uncertain ground truth: a case study in dermatologycited by 7
  4. 2024A Comprehensive Survey and Classification of Evaluation Criteria for Trustworthy Artificial Intelligencecited by 3
  5. 2023Investigating Reproducibility at Interspeech Conferences: A Longitudinal and Comparative Perspectivecited by 0
  6. 2023With a Little Help from the Authors: Reproducing Human Evaluation of an MT Error Detectorcited by 0
  7. 2026Who Annotates in NLP? A Large-scale Assessment of Human Annotation Reporting between 2018 and 2025cited by 0
  8. 2026Judgment-Grounded Expansion for Peer Review Generationcited by 0
  9. 2026RECLAIM: Can Agents Reproduce the Claims of Machine Learning Papers?cited by 0