# Deconstructing NLG Evaluation: Evaluation Practices, Assumptions, and Their Implications

arXiv: https://arxiv.org/abs/2205.06828 · HTML: https://mlsanity.com/paper/2205.06828/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 73 references that are in the map.
- First posted on arXiv 13 May 2022.

## Builds on (most cited first, 17 of 17)

- [Ethical and social risks of harm from Language Models](https://mlsanity.com/paper/2112.04359/) (2021)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus](https://mlsanity.com/paper/2104.08758/) (2021)
- [Measurement and Fairness](https://mlsanity.com/paper/1912.05511/) (2019)
- [All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text](https://mlsanity.com/paper/2107.00061/) (2021)
- [The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics](https://mlsanity.com/paper/2102.01672/) (2021)
- [Large image datasets: A pyrrhic win for computer vision?](https://mlsanity.com/paper/2006.16923/) (2020)
- [Why We Need New Evaluation Metrics for NLG](https://mlsanity.com/paper/1707.06875/) (2017)
- [Societal Biases in Language Generation: Progress and Challenges](https://mlsanity.com/paper/2105.04054/) (2021)
- [A Survey of Evaluation Metrics Used for NLG Systems](https://mlsanity.com/paper/2008.12009/) (2020)
- [The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation](https://mlsanity.com/paper/2109.06835/) (2021)
- [Lower Perplexity is Not Always Human-Like](https://mlsanity.com/paper/2106.01229/) (2021)
- [Perturbation CheckLists for Evaluating NLG Evaluation Metrics](https://mlsanity.com/paper/2109.05771/) (2021)
- [Overcoming Failures of Imagination in AI Infused System Development and Deployment](https://mlsanity.com/paper/2011.13416/) (2020)
- [Underreporting of errors in NLG output, and what to do about it](https://mlsanity.com/paper/2108.01182/) (2021)
- [Alexa, Google, Siri: What are Your Pronouns? Gender and Anthropomorphism in the Design and Perception of Conversational Assistants](https://mlsanity.com/paper/2106.02578/) (2021)
- [Towards Standard Criteria for human evaluation of Chatbots: A Survey](https://mlsanity.com/paper/2105.11197/) (2021)

## Built on by (most cited first, 30 of 32)

- [LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Texts](https://mlsanity.com/paper/2501.00274/) (2024)
- [Rethinking Model Evaluation as Narrowing the Socio-Technical Gap](https://mlsanity.com/paper/2306.03100/) (2023)
- [Evaluating Evaluation Metrics: A Framework for Analyzing NLG Evaluation Metrics using Measurement Theory](https://mlsanity.com/paper/2305.14889/) (2023)
- [Neither Valid nor Reliable? Investigating the Use of LLMs as Judges](https://mlsanity.com/paper/2508.18076/) (2025)
- [It Takes Two to Tango: Navigating Conceptualizations of NLP Tasks and Measurements of Performance](https://mlsanity.com/paper/2305.09022/) (2023)
- [Position: Measure Dataset Diversity, Don't Just Claim It](https://mlsanity.com/paper/2407.08188/) (2024)
- [The Authenticity Gap in Human Evaluation](https://mlsanity.com/paper/2205.11930/) (2022)
- [How to Select Datapoints for Efficient Human Evaluation of NLG Models?](https://mlsanity.com/paper/2501.18251/) (2025)
- [Automatic Metrics in Natural Language Generation: A Survey of Current Evaluation Practices](https://mlsanity.com/paper/2408.09169/) (2024)
- [Themis: A Reference-free NLG Evaluation Language Model with Flexibility and Interpretability](https://mlsanity.com/paper/2406.18365/) (2024)
- [METAL: Towards Multilingual Meta-Evaluation](https://mlsanity.com/paper/2404.01667/) (2024)
- [Aligning Offline Metrics and Human Judgments of Value for Code Generation Models](https://mlsanity.com/paper/2210.16494/) (2022)
- [When Personalization Meets Reality: A Multi-Faceted Analysis of Personalized Preference Learning](https://mlsanity.com/paper/2502.19158/) (2025)
- [Towards Robust NLG Bias Evaluation with Syntactically-diverse Prompts](https://mlsanity.com/paper/2212.01700/) (2022)
- [Responsible AI Research Needs Impact Statements Too](https://mlsanity.com/paper/2311.11776/) (2023)
- [Rigor in AI: Doing Rigorous AI Work Requires a Broader, Responsible AI-Informed Conception of Rigor](https://mlsanity.com/paper/2506.14652/) (2025)
- [Defining and Detecting Vulnerability in Human Evaluation Guidelines: A Preliminary Study Towards Reliable NLG Evaluation](https://mlsanity.com/paper/2406.07935/) (2024)
- [Contextual Metric Meta-Evaluation by Measuring Local Metric Accuracy](https://mlsanity.com/paper/2503.19828/) (2025)
- [Understanding and Meeting Practitioner Needs When Measuring Representational Harms Caused by LLM-Based Systems](https://mlsanity.com/paper/2506.04482/) (2025)
- [Responsible AI Considerations in Text Summarization Research: A Review of Current Practices](https://mlsanity.com/paper/2311.11103/) (2023)
- [Reassessing Active Learning Adoption in Contemporary NLP: A Community Survey](https://mlsanity.com/paper/2503.09701/) (2025)
- [From Queries to Criteria: Understanding How Astronomers Evaluate LLMs](https://mlsanity.com/paper/2507.15715/) (2025)
- [NLG Evaluation: Past, Present, Future](https://mlsanity.com/paper/2605.23715/) (2026)
- [LUNA: A Framework for Language Understanding and Naturalness Assessment](https://mlsanity.com/paper/2401.04522/) (2024)
- [From Lemmas to Dependencies: What Signals Drive Light Verbs Classification?](https://mlsanity.com/paper/2602.04127/) (2026)
- [From Understanding to Generation: An Efficient Shortcut for Evaluating Language Models](https://mlsanity.com/paper/2506.03592/) (2025)
- [A Conceptual Framework for AI Capability Evaluations](https://mlsanity.com/paper/2506.18213/) (2025)
- [AlignCheck: a Semantic Open-Domain Metric for Factual Consistency Assessment](https://mlsanity.com/paper/2512.03634/) (2025)
- [Results-Actionability Gap: Understanding How Practitioners Evaluate LLM Products in the Wild](https://mlsanity.com/paper/2604.16304/) (2026)
- [Supervision versus Demonstration-Based In-Context Learning for Multiword Expression Classification](https://mlsanity.com/paper/2606.07479/) (2026)

## Abstract (arXiv)

There are many ways to express similar things in text, which makes evaluating natural language generation (NLG) systems difficult. Compounding this difficulty is the need to assess varying quality criteria depending on the deployment setting. While the landscape of NLG evaluation has been well-mapped, practitioners' goals, assumptions, and constraints -- which inform decisions about what, when, and how to evaluate -- are often partially or implicitly stated, or not stated at all. Combining a formative semi-structured interview study of NLG practitioners (N=18) with a survey study of a broader sample of practitioners (N=61), we surface goals, community practices, assumptions, and constraints that shape NLG evaluations, examining their implications and how they embody ethical considerations.