mlsanityData to 14 Oct 2026

Deconstructing NLG Evaluation: Evaluation Practices, Assumptions, and Their Implications

arXiv 2205.06828 ↗First posted 13 May 2022cs.CL cs.AIcited by 32

Not on the map. Why?

Abstract (arXiv)

There are many ways to express similar things in text, which makes evaluating natural language generation (NLG) systems difficult. Compounding this difficulty is the need to assess varying quality criteria depending on the deployment setting. While the landscape of NLG evaluation has been well-mapped, practitioners' goals, assumptions, and constraints -- which inform decisions about what, when, and how to evaluate -- are often partially or implicitly stated, or not stated at all. Combining a formative semi-structured interview study of NLG practitioners (N=18) with a survey study of a broader sample of practitioners (N=61), we surface goals, community practices, assumptions, and constraints that shape NLG evaluations, examining their implications and how they embody ethical considerations.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Builds on 17 of its 73 references that are in the map. First posted on arXiv 13 May 2022.

Builds on

17 of 73 references.

References · 17

  1. 2021Ethical and social risks of harm from Language Modelscited by 966
  2. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 956
  3. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 492
  4. 2019Measurement and Fairnesscited by 308
  5. 2021All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Textcited by 287
  6. 2021The GEM Benchmark: Natural Language Generation, its Evaluation and Metricscited by 239
  7. 2020Large image datasets: A pyrrhic win for computer vision?cited by 237
  8. 2017Why We Need New Evaluation Metrics for NLGcited by 233
  9. 2021Societal Biases in Language Generation: Progress and Challengescited by 202
  10. 2020A Survey of Evaluation Metrics Used for NLG Systemscited by 164
  11. 2021The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generationcited by 97
  12. 2021Lower Perplexity is Not Always Human-Likecited by 52
5 more
  1. 2021Perturbation CheckLists for Evaluating NLG Evaluation Metricscited by 50
  2. 2020Overcoming Failures of Imagination in AI Infused System Development and Deploymentcited by 22
  3. 2021Underreporting of errors in NLG output, and what to do about itcited by 19
  4. 2021Alexa, Google, Siri: What are Your Pronouns? Gender and Anthropomorphism in the Design and Perception of Conversational Assistantscited by 16
  5. 2021Towards Standard Criteria for human evaluation of Chatbots: A Surveycited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 5; 2024: 7; 2025: 11; 2026: 6.

32 papers.

Sort:

By area:

  1. 2024LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Textscited by 87
  2. 2023Rethinking Model Evaluation as Narrowing the Socio-Technical Gapcited by 56
  3. 2023Evaluating Evaluation Metrics: A Framework for Analyzing NLG Evaluation Metrics using Measurement Theorycited by 43
  4. 2025Neither Valid nor Reliable? Investigating the Use of LLMs as Judgescited by 39
  5. 2023It Takes Two to Tango: Navigating Conceptualizations of NLP Tasks and Measurements of Performancecited by 26
  6. 2024Position: Measure Dataset Diversity, Don't Just Claim Itcited by 26
  7. 2022The Authenticity Gap in Human Evaluationcited by 24
  8. 2025How to Select Datapoints for Efficient Human Evaluation of NLG Models?cited by 23
  9. 2024Automatic Metrics in Natural Language Generation: A Survey of Current Evaluation Practicescited by 20
  10. 2024Themis: A Reference-free NLG Evaluation Language Model with Flexibility and Interpretabilitycited by 19
  11. 2024METAL: Towards Multilingual Meta-Evaluationcited by 15
  12. 2022Aligning Offline Metrics and Human Judgments of Value for Code Generation Modelscited by 14
20 more
  1. 2025When Personalization Meets Reality: A Multi-Faceted Analysis of Personalized Preference Learningcited by 9
  2. 2022Towards Robust NLG Bias Evaluation with Syntactically-diverse Promptscited by 7
  3. 2023Responsible AI Research Needs Impact Statements Toocited by 7
  4. 2025Rigor in AI: Doing Rigorous AI Work Requires a Broader, Responsible AI-Informed Conception of Rigorcited by 7
  5. 2024Defining and Detecting Vulnerability in Human Evaluation Guidelines: A Preliminary Study Towards Reliable NLG Evaluationcited by 5
  6. 2025Contextual Metric Meta-Evaluation by Measuring Local Metric Accuracycited by 3
  7. 2025Understanding and Meeting Practitioner Needs When Measuring Representational Harms Caused by LLM-Based Systemscited by 3
  8. 2023Responsible AI Considerations in Text Summarization Research: A Review of Current Practicescited by 2
  9. 2025Reassessing Active Learning Adoption in Contemporary NLP: A Community Surveycited by 2
  10. 2025From Queries to Criteria: Understanding How Astronomers Evaluate LLMscited by 2
  11. 2026NLG Evaluation: Past, Present, Futurecited by 2
  12. 2024LUNA: A Framework for Language Understanding and Naturalness Assessmentcited by 1
  13. 2026From Lemmas to Dependencies: What Signals Drive Light Verbs Classification?cited by 1
  14. 2025From Understanding to Generation: An Efficient Shortcut for Evaluating Language Modelscited by 0
  15. 2025A Conceptual Framework for AI Capability Evaluationscited by 0
  16. 2025AlignCheck: a Semantic Open-Domain Metric for Factual Consistency Assessmentcited by 0
  17. 2026Results-Actionability Gap: Understanding How Practitioners Evaluate LLM Products in the Wildcited by 0
  18. 2026Supervision versus Demonstration-Based In-Context Learning for Multiword Expression Classificationcited by 0
  19. 2026Litmus: Zero-Label, Code-Driven Metric Specification for Evaluating AI Systemscited by 0
  20. 2026LLJ Cards: Best practices for the Use of LLMs as Judgescited by 0