mlsanityData to 14 Oct 2026

The Great Misalignment Problem in Human Evaluation of NLP Methods

arXiv 2104.05361 ↗First posted 12 Apr 2021cs.CLcited by 8

Not on the map. Why?

Abstract (arXiv)

We outline the Great Misalignment Problem in natural language processing research, this means simply that the problem definition is not in line with the method proposed and the human evaluation is not in line with the definition nor the method. We study this misalignment problem by surveying 10 randomly sampled papers published in ACL 2020 that report results with human evaluation. Our results show that only one paper was fully in line in terms of problem definition, method and evaluation. Only two papers presented a human evaluation that was in line with what was modeled in the method. These results highlight that the Great Misalignment Problem is a major one and it affects the validity and reproducibility of results obtained by a human evaluation.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 6 of its 29 references that are in the map. First posted on arXiv 12 Apr 2021.

Builds on

6 of 29 references.

References · 6

  1. 2017Why We Need New Evaluation Metrics for NLGcited by 233
  2. 2020Towards Transparent and Explainable Attention Modelscited by 47
  3. 2019Let's FACE it. Finnish Poetry Generation with Aesthetics and Framingcited by 11
  4. 2020Automatic Dialect Adaptation in Finnish and its Effect on Perceived Creativitycited by 5
  5. 2019Modelling the Socialization of Creative Agents in a Master-Apprentice Setting: The Case of Movie Title Punscited by 3
  6. 2019The University of Helsinki submissions to the WMT19 news translation taskcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 2; 2023: 0; 2024: 2; 2025: 1; 2026: 1.

8 papers.

  1. 2021A Review of Human Evaluation for Style Transfercited by 19
  2. 2021Human Evaluation of Creative NLG Systems: An Interdisciplinary Survey on Recent Paperscited by 18
  3. 2024AGENT-CQ: Automatic Generation and Evaluation of Clarifying Questions for Conversational Search with LLMscited by 7
  4. 2026Evaluating Human-AI Safety: A Framework for Measuring Harmful Capability Upliftcited by 3
  5. 2022Automatic Generation of Factual News Headlines in Finnishcited by 1
  6. 2022Modern French Poetry Generation with RoBERTa and GPT-2cited by 1
  7. 2024A Perspective on Literary Metaphor in the Context of Generative AIcited by 1
  8. 2025From NLG Evaluation to Modern Student Assessment in the Era of ChatGPT: The Great Misalignment Problem and Pedagogical Multi-Factor Assessment (P-MFA)cited by 0