# Missing Information, Unresponsive Authors, Experimental Flaws: The Impossibility of Assessing the Reproducibility of Previous Human Evaluations in NLP

arXiv: https://arxiv.org/abs/2305.01633 · HTML: https://mlsanity.com/paper/2305.01633/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 2 of its 29 references that are in the map.
- First posted on arXiv 2 May 2023.

## Builds on (most cited first, 2 of 2)

- [The Human Evaluation Datasheet 1.0: A Template for Recording Details of Human Evaluation Experiments in NLP](https://mlsanity.com/paper/2103.09710/) (2021)
- [Quantified Reproducibility Assessment of NLP Results](https://mlsanity.com/paper/2204.05961/) (2022)

## Built on by (most cited first, 21 of 21)

- [Humans or LLMs as the Judge? A Study on Judgement Biases](https://mlsanity.com/paper/2402.10669/) (2024)
- [KoLA: Carefully Benchmarking World Knowledge of Large Language Models](https://mlsanity.com/paper/2306.09296/) (2023)
- [GPTEval: A Survey on Assessments of ChatGPT and GPT-4](https://mlsanity.com/paper/2308.12488/) (2023)
- [Are LLM-Judges Robust to Expressions of Uncertainty? Investigating the effect of Epistemic Markers on LLM-based Evaluation](https://mlsanity.com/paper/2410.20774/) (2024)
- [How Good Are LLMs for Literary Translation, Really? Literary Translation Evaluation with Humans and LLMs](https://mlsanity.com/paper/2410.18697/) (2024)
- [Automatic Metrics in Natural Language Generation: A Survey of Current Evaluation Practices](https://mlsanity.com/paper/2408.09169/) (2024)
- [Beyond Probabilities: Unveiling the Misalignment in Evaluating Large Language Models](https://mlsanity.com/paper/2402.13887/) (2024)
- [Understanding Counterspeech for Online Harm Mitigation](https://mlsanity.com/paper/2307.04761/) (2023)
- [Evaluating Diversity in Automatic Poetry Generation](https://mlsanity.com/paper/2406.15267/) (2024)
- [Verifiable evaluations of machine learning models using zkSNARKs](https://mlsanity.com/paper/2402.02675/) (2024)
- [The Promises and Pitfalls of LLM Annotations in Dataset Labeling: a Case Study on Media Bias Detection](https://mlsanity.com/paper/2411.11081/) (2024)
- [Evaluating Task-oriented Dialogue Systems: A Systematic Review of Measures, Constructs and their Operationalisations](https://mlsanity.com/paper/2312.13871/) (2023)
- [SPHERE: An Evaluation Card for Human-AI Systems](https://mlsanity.com/paper/2504.07971/) (2025)
- [Cross-lingual Cross-temporal Summarization: Dataset, Models, Evaluation](https://mlsanity.com/paper/2306.12916/) (2023)
- [Evaluating AI systems under uncertain ground truth: a case study in dermatology](https://mlsanity.com/paper/2307.02191/) (2023)
- [A Comprehensive Survey and Classification of Evaluation Criteria for Trustworthy Artificial Intelligence](https://mlsanity.com/paper/2410.17281/) (2024)
- [Who Annotates in NLP? A Large-scale Assessment of Human Annotation Reporting between 2018 and 2025](https://mlsanity.com/paper/2606.02255/) (2026)
- [Investigating Reproducibility at Interspeech Conferences: A Longitudinal and Comparative Perspective](https://mlsanity.com/paper/2306.10033/) (2023)
- [With a Little Help from the Authors: Reproducing Human Evaluation of an MT Error Detector](https://mlsanity.com/paper/2308.06527/) (2023)
- [Judgment-Grounded Expansion for Peer Review Generation](https://mlsanity.com/paper/2606.23233/) (2026)
- [RECLAIM: Can Agents Reproduce the Claims of Machine Learning Papers?](https://mlsanity.com/paper/2609.28850/) (2026)

## Abstract (arXiv)

We report our efforts in identifying a set of previous human evaluations in NLP that would be suitable for a coordinated study examining what makes human evaluations in NLP more/less reproducible. We present our results and findings, which include that just 13% of papers had (i) sufficiently low barriers to reproduction, and (ii) enough obtainable information, to be considered for reproduction, and that all but one of the experiments we selected for reproduction was discovered to have flaws that made the meaningfulness of conducting a reproduction questionable. As a result, we had to change our coordinated study design from a reproduce approach to a standardise-then-reproduce-twice approach. Our overall (negative) finding that the great majority of human evaluations in NLP is not repeatable and/or not reproducible and/or too flawed to justify reproduction, paints a dire picture, but presents an opportunity for a rethink about how to design and report human evaluations in NLP.