mlsanityData to 7 Oct 2026

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text

arXiv 2202.06935 ↗First posted 14 Feb 2022cs.CL cs.AI cs.LGcited by 154

Not on the map. Why?

Abstract (arXiv)

Evaluation practices in natural language generation (NLG) have many known flaws, but improved evaluation approaches are rarely widely adopted. This issue has become more urgent, since neural NLG models have improved to the point where they can often no longer be distinguished based on the surface-level features that older metrics rely on. This paper surveys the issues with human and automatic model evaluations and with commonly used datasets in NLG that have been pointed out over the past 20 years. We summarize, categorize, and discuss how researchers have been addressing these issues and what their findings mean for the current state of model evaluations. Building on those insights, we lay out a long-term vision for NLG evaluation and propose concrete steps for researchers to improve their evaluation processes. Finally, we analyze 66 NLG papers from recent NLP conferences in how well they already follow these suggestions and identify which areas require more drastic changes to the status quo.

Key facts. Cited by 154 papers in the map (data through 7 Oct 2026). Builds on 156 of its 304 references that are in the map. First posted on arXiv 14 Feb 2022.

Builds on

156 of 304 references.

References · 156

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  5. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  6. 2020Learning to summarize from human feedbackcited by 2,562
  7. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  8. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  9. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,400
  10. 2018Datasheets for Datasetscited by 1,371
  11. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  12. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
38 more
  1. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  2. 2018Model Cards for Model Reportingcited by 1,068
  3. 2015Teaching Machines to Read and Comprehendcited by 1,039
  4. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  5. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  6. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  7. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  8. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  9. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  10. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  11. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  12. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  13. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  14. 2018MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modellingcited by 760
  15. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  16. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  17. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 578
  18. 2019Defending Against Neural Fake Newscited by 553
  19. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
  20. 2021Datasets: A Community Library for Natural Language Processingcited by 526
  21. 2019GLTR: Statistical Detection and Visualization of Generated Textcited by 490
  22. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  23. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
  24. 2021Dynabench: Rethinking Benchmarking in NLPcited by 447
  25. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 415
  26. 2015A Neural Attention Model for Abstractive Sentence Summarizationcited by 394
  27. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 376
  28. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 364
  29. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 346
  30. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 333
  31. 2018Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategiescited by 292
  32. 2020FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarizationcited by 292
  33. 2021All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Textcited by 287
  34. 2021Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metricscited by 285
  35. 2018Analysis Methods in Neural Language Processing: A Surveycited by 276
  36. 2020Evaluation of Text Generation: A Surveycited by 275
  37. 2020ToTTo: A Controlled Table-To-Text Generation Datasetcited by 267
  38. 2019Automatic Detection of Generated Text is Easiest when Humans are Fooledcited by 260

50 most cited of 156 All 156 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 30; 2023: 40; 2024: 39; 2025: 29; 2026: 16.

154 papers.

Sort:

By area:

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  3. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  4. 2023BloombergGPT: A Large Language Model for Financecited by 923
  5. 2022News Summarization and Evaluation in the Era of GPT-3cited by 324
  6. 2022UL2: Unifying Language Learning Paradigmscited by 292
  7. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
  8. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 252
  9. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
  10. 2023Does Writing with Language Models Reduce Content Diversity?cited by 190
  11. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 174
  12. 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172
38 more
  1. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 141
  2. 2024Cooperate or Collapse: Emergence of Sustainable Cooperation in a Society of LLM Agentscited by 120
  3. 2023AI Transparency in the Age of LLMs: A Human-Centered Research Roadmapcited by 119
  4. 2023A Critical Evaluation of Evaluations for Long-form Question Answeringcited by 111
  5. 2023LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarizationcited by 96
  6. 2023Evaluating Language-Model Agents on Realistic Autonomous Taskscited by 94
  7. 2023Large language models effectively leverage document-level context for literary translation, but critical errors persistcited by 80
  8. 2023Large Language Models Suffer From Their Own Output: An Analysis of the Self-Consuming Training Loopcited by 75
  9. 2023EvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteriacited by 73
  10. 2022RankGen: Improving Text Generation with Large Ranking Modelscited by 72
  11. 2024LLM-based NLG Evaluation: Current Status and Challengescited by 71
  12. 2023Uncertainty in Natural Language Generation: From Theory to Applicationscited by 61
  13. 2025Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluationcited by 57
  14. 2023Rethinking Model Evaluation as Narrowing the Socio-Technical Gapcited by 56
  15. 2023Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generationcited by 55
  16. 2022Follow the Wisdom of the Crowd: Effective Text Generation via Minimum Bayes Risk Decodingcited by 54
  17. 2024Quantifying Variance in Evaluation Benchmarkscited by 52
  18. 2023What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variabilitycited by 51
  19. 2022SQuALITY: Building a Long-Document Summarization Dataset the Hard Waycited by 50
  20. 2022Prompted Opinion Summarization with GPT-3.5cited by 44
  21. 2022mFACE: Multilingual Summarization with Factual Consistency Evaluationcited by 42
  22. 2023SEAHORSE: A Multilingual, Multifaceted Dataset for Summarization Evaluationcited by 42
  23. 2022SafeText: A Benchmark for Exploring Physical Safety in Language Modelscited by 41
  24. 2023Evaluating Evaluation Metrics: A Framework for Analyzing NLG Evaluation Metrics using Measurement Theorycited by 41
  25. 2022GEMv2: Multilingual NLG Benchmarking in a Single Line of Codecited by 36
  26. 2022DEMETR: Diagnosing Evaluation Metrics for Translationcited by 36
  27. 2024AI Hallucinations: A Misnomer Worth Clarifyingcited by 34
  28. 2024K-QA: A Real-World Medical Q&A Benchmarkcited by 30
  29. 2022Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarizationcited by 29
  30. 2023Benchmarking Large Language Model Capabilities for Conditional Generationcited by 29
  31. 2022deep-significance - Easy and Meaningful Statistical Significance Testing in the Age of Neural Networkscited by 28
  32. 2023X-Eval: Generalizable Multi-aspect Text Evaluation via Augmented Instruction Tuning with Auxiliary Evaluation Aspectscited by 28
  33. 2024Data-Centric Foundation Models in Computational Healthcare: A Surveycited by 27
  34. 2024Fine-Grained Natural Language Inference Based Faithfulness Evaluation for Diverse Summarisation Taskscited by 23
  35. 2024Grounding and Evaluation for Large Language Models: Practical Challenges and Lessons Learned (Survey)cited by 20
  36. 2023Stealing the Decoding Algorithms of Language Modelscited by 19
  37. 2024SPADE: Synthesizing Data Quality Assertions for Large Language Model Pipelinescited by 19
  38. 2024Recent Trends in Personalized Dialogue Generation: A Review of Datasets, Methodologies, and Evaluationscited by 19

50 most cited of 154 All 154 (JSON)