# The Great Misalignment Problem in Human Evaluation of NLP Methods

arXiv: https://arxiv.org/abs/2104.05361 · HTML: https://mlsanity.com/paper/2104.05361/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 29 references that are in the map.
- First posted on arXiv 12 Apr 2021.

## Builds on (most cited first, 6 of 6)

- [Why We Need New Evaluation Metrics for NLG](https://mlsanity.com/paper/1707.06875/) (2017)
- [Towards Transparent and Explainable Attention Models](https://mlsanity.com/paper/2004.14243/) (2020)
- [Let's FACE it. Finnish Poetry Generation with Aesthetics and Framing](https://mlsanity.com/paper/1910.13946/) (2019)
- [Automatic Dialect Adaptation in Finnish and its Effect on Perceived Creativity](https://mlsanity.com/paper/2009.02685/) (2020)
- [Modelling the Socialization of Creative Agents in a Master-Apprentice Setting: The Case of Movie Title Puns](https://mlsanity.com/paper/1907.04954/) (2019)
- [The University of Helsinki submissions to the WMT19 news translation task](https://mlsanity.com/paper/1906.04040/) (2019)

## Built on by (most cited first, 8 of 8)

- [A Review of Human Evaluation for Style Transfer](https://mlsanity.com/paper/2106.04747/) (2021)
- [Human Evaluation of Creative NLG Systems: An Interdisciplinary Survey on Recent Papers](https://mlsanity.com/paper/2108.00308/) (2021)
- [AGENT-CQ: Automatic Generation and Evaluation of Clarifying Questions for Conversational Search with LLMs](https://mlsanity.com/paper/2410.19692/) (2024)
- [Evaluating Human-AI Safety: A Framework for Measuring Harmful Capability Uplift](https://mlsanity.com/paper/2603.26676/) (2026)
- [Automatic Generation of Factual News Headlines in Finnish](https://mlsanity.com/paper/2212.02170/) (2022)
- [Modern French Poetry Generation with RoBERTa and GPT-2](https://mlsanity.com/paper/2212.02911/) (2022)
- [A Perspective on Literary Metaphor in the Context of Generative AI](https://mlsanity.com/paper/2409.01053/) (2024)
- [From NLG Evaluation to Modern Student Assessment in the Era of ChatGPT: The Great Misalignment Problem and Pedagogical Multi-Factor Assessment (P-MFA)](https://mlsanity.com/paper/2512.15183/) (2025)

## Abstract (arXiv)

We outline the Great Misalignment Problem in natural language processing research, this means simply that the problem definition is not in line with the method proposed and the human evaluation is not in line with the definition nor the method. We study this misalignment problem by surveying 10 randomly sampled papers published in ACL 2020 that report results with human evaluation. Our results show that only one paper was fully in line in terms of problem definition, method and evaluation. Only two papers presented a human evaluation that was in line with what was modeled in the method. These results highlight that the Great Misalignment Problem is a major one and it affects the validity and reproducibility of results obtained by a human evaluation.