mlsanityData to 7 Oct 2026

Why We Need New Evaluation Metrics for NLG

arXiv 1707.06875 ↗First posted 21 Jul 2017cs.CLcited by 232

Not on the map. Why?

Abstract (arXiv)

The majority of NLG evaluation relies on automatic metrics, such as BLEU . In this paper, we motivate the need for novel, system- and data-independent automatic evaluation methods: We investigate a wide range of metrics, including state-of-the-art word-based and novel grammar-based ones, and demonstrate that they only weakly reflect human judgements of system outputs as generated by data-driven, end-to-end NLG. We also show that metric performance is data- and system-specific. Nevertheless, our results also suggest that automatic metrics perform reliably at system-level and can support system development by finding cases where a system performs poorly.

Key facts. Cited by 232 papers in the map (data through 7 Oct 2026). First posted on arXiv 21 Jul 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 55; 2022: 44; 2023: 34; 2024: 40; 2025: 31; 2026: 27.

232 papers.

Sort:

By area:

  1. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  2. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  3. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  4. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
  5. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 367
  6. 2021All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Textcited by 287
  7. 2023Can Large Language Models Transform Computational Social Science?cited by 245
  8. 2021QuestEval: Summarization Asks for Fact-based Evaluationcited by 241
  9. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  10. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 138
  11. 2021Advances and Challenges in Conversational Recommender Systems: A Surveycited by 118
  12. 2021A Comprehensive Assessment of Dialog Evaluation Metricscited by 90
38 more
  1. 2024CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklistscited by 87
  2. 2023Human Feedback is not Gold Standardcited by 82
  3. 2022Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generationcited by 73
  4. 2024SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Wordscited by 67
  5. 2020Text Generation by Learning from Demonstrationscited by 64
  6. 2021Building and Evaluating Open-Domain Dialogue Corpora with Clarifying Questionscited by 63
  7. 2021OpenMEVA: A Benchmark for Evaluating Open-ended Story Generation Metricscited by 61
  8. 2024Understanding LLMs: A Comprehensive Overview from Training to Inferencecited by 58
  9. 2021IndoNLG: Benchmark and Resources for Evaluating Indonesian Natural Language Generationcited by 57
  10. 2021A Systematic Investigation of Commonsense Knowledge in Large Language Modelscited by 56
  11. 2021MERMAID: Metaphor Generation with Symbolism and Discriminative Decodingcited by 49
  12. 2021Shades of BLEU, Flavours of Success: The Case of MultiWOZcited by 48
  13. 2021"Let Your Characters Tell Their Story": A Dataset for Character-Centric Narrative Understandingcited by 48
  14. 2021Controlling Hallucinations at Word Level in Data-to-Text Generationcited by 41
  15. 2023Evaluating Evaluation Metrics: A Framework for Analyzing NLG Evaluation Metrics using Measurement Theorycited by 41
  16. 2021It's not Rocket Science : Interpreting Figurative Language in Narrativescited by 39
  17. 2024KIEval: A Knowledge-grounded Interactive Evaluation Framework for Large Language Modelscited by 36
  18. 2023GameEval: Evaluating LLMs on Conversational Gamescited by 35
  19. 2021A Study of Automatic Metrics for the Evaluation of Natural Language Explanationscited by 34
  20. 2022MAUVE Scores for Generative Models: Theory and Practicecited by 34
  21. 2024LLM Harmony: Multi-Agent Communication for Problem Solvingcited by 34
  22. 2021Can Audio Captions Be Evaluated with Image Caption Metrics?cited by 33
  23. 2021PPL-MCTS: Constrained Textual Generation Through Discriminator-Guided MCTS Decodingcited by 32
  24. 2022Deconstructing NLG Evaluation: Evaluation Practices, Assumptions, and Their Implicationscited by 32
  25. 2023Foundation Metrics for Evaluating Effectiveness of Healthcare Conversations Powered by Generative AIcited by 32
  26. 2024Do Language Models Enjoy Their Own Stories? Prompting Large Language Models for Automatic Story Evaluationcited by 32
  27. 2022Using Pre-Trained Language Models for Producing Counter Narratives Against Hate Speech: a Comparative Studycited by 30
  28. 2023Distilled GPT for Source Code Summarizationcited by 30
  29. 2021OTTers: One-turn Topic Transitions for Open-Domain Dialoguecited by 29
  30. 2025LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs -- No Silver Bullet for LC or RAG Routingcited by 29
  31. 2021Reliability Testing for Natural Language Processing Systemscited by 25
  32. 2024Automated Evaluation of Retrieval-Augmented Language Models with Task-Specific Exam Generationcited by 25
  33. 2022Evaluating Mixed-initiative Conversational Search Systems via User Simulationcited by 24
  34. 2024Stratified Prediction-Powered Inference for Hybrid Language Model Evaluationcited by 24
  35. 2025The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMscited by 24
  36. 2021Logic-Consistency Text Generation from Semantic Parsescited by 23
  37. 2022Automated Audio Captioning: An Overview of Recent Progress and New Challengescited by 22
  38. 2023What's the Meaning of Superhuman Performance in Today's NLU?cited by 22

50 most cited of 232 All 232 (JSON)