# Human Evaluation of Creative NLG Systems: An Interdisciplinary Survey on Recent Papers

arXiv: https://arxiv.org/abs/2108.00308 · HTML: https://mlsanity.com/paper/2108.00308/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 72 references that are in the map.
- First posted on arXiv 31 Jul 2021.

## Builds on (most cited first, 14 of 14)

- [Deep Reinforcement Learning for Dialogue Generation](https://mlsanity.com/paper/1606.01541/) (2016)
- [The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics](https://mlsanity.com/paper/2102.01672/) (2021)
- [Chart-to-Text: Generating Natural Language Descriptions for Charts by Adapting the Transformer Model](https://mlsanity.com/paper/2010.09142/) (2020)
- [Attaining the Unattainable? Reassessing Claims of Human Parity in Neural Machine Translation](https://mlsanity.com/paper/1808.10432/) (2018)
- [ReviewRobot: Explainable Paper Review Generation based on Knowledge Synthesis](https://mlsanity.com/paper/2010.06119/) (2020)
- [GLGE: A New General Language Generation Evaluation Benchmark](https://mlsanity.com/paper/2011.11928/) (2020)
- [Stable Style Transformer: Delete and Generate Approach with Encoder-Decoder for Text Style Transfer](https://mlsanity.com/paper/2005.12086/) (2020)
- [Policy-Driven Neural Response Generation for Knowledge-Grounded Dialogue Systems](https://mlsanity.com/paper/2005.12529/) (2020)
- [Toward Automated Quest Generation in Text-Adventure Games](https://mlsanity.com/paper/1909.06283/) (2019)
- [Let's FACE it. Finnish Poetry Generation with Aesthetics and Framing](https://mlsanity.com/paper/1910.13946/) (2019)
- [Paranoid Transformer: Reading Narrative of Madness as Computational Approach to Creativity](https://mlsanity.com/paper/2007.06290/) (2020)
- [The Great Misalignment Problem in Human Evaluation of NLP Methods](https://mlsanity.com/paper/2104.05361/) (2021)
- [Automatic Dialect Adaptation in Finnish and its Effect on Perceived Creativity](https://mlsanity.com/paper/2009.02685/) (2020)
- [Shimon the Rapper: A Real-Time System for Human-Robot Interactive Rap Battles](https://mlsanity.com/paper/2009.09234/) (2020)

## Built on by (most cited first, 18 of 18)

- [LLM-based NLG Evaluation: Current Status and Challenges](https://mlsanity.com/paper/2402.01383/) (2024)
- [On the Blind Spots of Model-Based Evaluation Metrics for Text Generation](https://mlsanity.com/paper/2212.10020/) (2022)
- [Pron vs Prompt: Can Large Language Models already Challenge a World-Class Fiction Author at Creative Text Writing?](https://mlsanity.com/paper/2407.01119/) (2024)
- [Automatic Metrics in Natural Language Generation: A Survey of Current Evaluation Practices](https://mlsanity.com/paper/2408.09169/) (2024)
- [Can ChatGPT Perform Reasoning Using the IRAC Method in Analyzing Legal Scenarios Like a Lawyer?](https://mlsanity.com/paper/2310.14880/) (2023)
- [What is Wrong with Language Models that Can Not Tell a Story?](https://mlsanity.com/paper/2211.05044/) (2022)
- [Post Turing: Mapping the landscape of LLM Evaluation](https://mlsanity.com/paper/2311.02049/) (2023)
- [CoCo Matrix: Taxonomy of Cognitive Contributions in Co-writing with Intelligent Agents](https://mlsanity.com/paper/2405.12438/) (2024)
- [A Comparative Analysis of Ethical and Safety Gaps in LLMs using Relative Danger Coefficient](https://mlsanity.com/paper/2505.04654/) (2025)
- [A Comprehensive Survey of Natural Language Generation Advances from the Perspective of Digital Deception](https://mlsanity.com/paper/2208.05757/) (2022)
- [Automatic Generation of Factual News Headlines in Finnish](https://mlsanity.com/paper/2212.02170/) (2022)
- [Modern French Poetry Generation with RoBERTa and GPT-2](https://mlsanity.com/paper/2212.02911/) (2022)
- [A Perspective on Literary Metaphor in the Context of Generative AI](https://mlsanity.com/paper/2409.01053/) (2024)
- [Erato: Automatizing Poetry Evaluation](https://mlsanity.com/paper/2310.20326/) (2023)
- [A Conceptual Framework for AI Capability Evaluations](https://mlsanity.com/paper/2506.18213/) (2025)
- [From NLG Evaluation to Modern Student Assessment in the Era of ChatGPT: The Great Misalignment Problem and Pedagogical Multi-Factor Assessment (P-MFA)](https://mlsanity.com/paper/2512.15183/) (2025)
- [Aggregate vs. Personalized Judges in Business Idea Evaluation: Evidence from Expert Disagreement](https://mlsanity.com/paper/2604.22517/) (2026)
- [Grounded Satirical Generation with RAG](https://mlsanity.com/paper/2605.10853/) (2026)

## Abstract (arXiv)

We survey human evaluation in papers presenting work on creative natural language generation that have been published in INLG 2020 and ICCC 2020. The most typical human evaluation method is a scaled survey, typically on a 5 point scale, while many other less common methods exist. The most commonly evaluated parameters are meaning, syntactic correctness, novelty, relevance and emotional value, among many others. Our guidelines for future evaluation include clearly defining the goal of the generative system, asking questions as concrete as possible, testing the evaluation setup, using multiple different evaluation setups, reporting the entire evaluation process and potential biases clearly, and finally analyzing the evaluation results in a more profound way than merely reporting the most typical statistics.