mlsanityData to 7 Oct 2026

Automatic Metrics in Natural Language Generation: A Survey of Current Evaluation Practices

arXiv 2408.09169 ↗First posted 17 Aug 2024cs.CLcited by 19

Not on the map. Why?

Abstract (arXiv)

Automatic metrics are extensively used to evaluate natural language processing systems. However, there has been increasing focus on how they are used and reported by practitioners within the field. In this paper, we have conducted a survey on the use of automatic metrics, focusing particularly on natural language generation (NLG) tasks. We inspect which metrics are used as well as why they are chosen and how their use is reported. Our findings from this survey reveal significant shortcomings, including inappropriate metric usage, lack of implementation details and missing correlations with human judgements. We conclude with recommendations that we believe authors should follow to enable more rigour within the field.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 72 of its 133 references that are in the map. First posted on arXiv 17 Aug 2024.

Builds on

72 of 133 references.

References · 72

  1. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  2. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  3. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  4. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  5. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  6. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  7. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 598
  8. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  9. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 484
  10. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
  11. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 415
  12. 2022RARR: Researching and Revising What Language Models Say, Using Language Modelscited by 286
38 more
  1. 2023AlignScore: Evaluating Factual Consistency with a Unified Alignment Functioncited by 284
  2. 2020Evaluation of Text Generation: A Surveycited by 275
  3. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
  4. 2017Why We Need New Evaluation Metrics for NLGcited by 232
  5. 2022SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Controlcited by 191
  6. 2022Distilling Reasoning Capabilities into Smaller Language Modelscited by 155
  7. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  8. 2021How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVENcited by 124
  9. 2023Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventionscited by 121
  10. 2019Handling Divergent Reference Texts when Evaluating Table-to-Text Generationcited by 119
  11. 2022Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacycited by 97
  12. 2019Do Massively Pretrained Language Models Make Better Storytellers?cited by 90
  13. 2022Efficient Transformers with Dynamic Token Poolingcited by 75
  14. 2023The Next Chapter: A Study of Large Language Models in Storytellingcited by 58
  15. 2022LENS: A Learnable Evaluation Metric for Text Simplificationcited by 54
  16. 2022On the Blind Spots of Model-Based Evaluation Metrics for Text Generationcited by 48
  17. 2022Critic-Guided Decoding for Controlled Text Generationcited by 45
  18. 2022Conditional Generation with a Question-Answering Blueprintcited by 44
  19. 2020NUBIA: NeUral Based Interchangeability Assessor for Text Generationcited by 34
  20. 2020A Gold Standard Methodology for Evaluating Accuracy in Data-To-Text Systemscited by 34
  21. 2022Deconstructing NLG Evaluation: Evaluation Practices, Assumptions, and Their Implicationscited by 32
  22. 2022CTRLEval: An Unsupervised Reference-Free Metric for Evaluating Controlled Text Generationcited by 29
  23. 2023PREADD: Prefix-Adaptive Decoding for Controlled Text Generationcited by 29
  24. 2022Controllable Text Generation via Probability Density Estimation in the Latent Spacecited by 27
  25. 2023BOLT: Fast Energy-based Controlled Text Generation with Tunable Biasescited by 24
  26. 2023Missing Information, Unresponsive Authors, Experimental Flaws: The Impossibility of Assessing the Reproducibility of Previous Human Evaluations in NLPcited by 21
  27. 2021Underreporting of errors in NLG output, and what to do about itcited by 19
  28. 2023Multi-source Semantic Graph-based Multimodal Sarcasm Explanation Generationcited by 19
  29. 2021Human Evaluation of Creative NLG Systems: An Interdisciplinary Survey on Recent Paperscited by 18
  30. 2023On the Efficacy of Sampling Adapterscited by 17
  31. 2023A Systematic Study of Knowledge Distillation for Natural Language Generation with Pseudo-Target Trainingcited by 16
  32. 2023NLG Evaluation Metrics Beyond Correlation Analysis: An Empirical Metric Preference Checklistcited by 16
  33. 2023TwistList: Resources and Baselines for Tongue Twister Generationcited by 16
  34. 2023Enhancing Dialogue Generation via Dynamic Graph Knowledge Aggregationcited by 16
  35. 2023This is not correct! Negation-aware Evaluation of Language Generation Systemscited by 16
  36. 2023Summaries as Captions: Generating Figure Captions for Scientific Documents with Automated Text Summarizationcited by 14
  37. 2022An Extensible Plug-and-Play Method for Multi-Aspect Controllable Text Generationcited by 10
  38. 2023Teaching the Pre-trained Model to Generate Simple Texts for Text Simplificationcited by 10

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 11.

19 papers.

Sort:

By area:

  1. 2025Revisiting Compositional Generalization Capability of Large Language Models Considering Instruction Following Abilitycited by 10
  2. 2025LLMs as Span Annotators: A Comparative Study of LLMs and Humanscited by 7
  3. 2025From Vision To Language through Graph of Events in Space and Time: An Explainable Self-supervised Approachcited by 4
  4. 2025References Matter: Investigating the Impact of Reference Set Variation on Summarization Evaluationcited by 2
  5. 2026Computational Compliance for AI Regulation: Blueprint for a New Research Domaincited by 1
  6. 2026Creativity Bias: How Machine Evaluation Struggles with Creativity in Literary Translationscited by 1
  7. 2026MATCHA: Matching Text via Contrastive Semantic Alignmentcited by 1
  8. 2025Natural Language Generationcited by 0
  9. 2025Real-World Summarization: When Evaluation Reaches Its Limitscited by 0
  10. 2025Can LLM-Generated Textual Explanations Enhance Model Classification Performance? An Empirical Studycited by 0
  11. 2025CLEAR: A Comprehensive Linguistic Evaluation of Argument Rewriting by Large Language Modelscited by 0
  12. 2026What Are We Measuring in NLG? A Meta-Analysis of Evaluation Trends 2020-2025cited by 0
7 more
  1. 2026Impact of enriched meaning representations for language generation in dialogue tasks: A comprehensive exploration of the relevance of tasks, corpora and metricscited by 0
  2. 2026Benchmarked Yet Not Measured -- Generative AI Should be Evaluated Against Real-World Utilitycited by 0
  3. 2026Litmus: Zero-Label, Code-Driven Metric Specification for Evaluating AI Systemscited by 0
  4. 2026SurveyReview: A Reviewer-Aligned Benchmark for Survey Evaluatorscited by 0
  5. 2026LANTERN: Language Model Assessment on Noisy and Transformed Tasks for Understanding Error and Robustness Nuancescited by 0
  6. 2026AgriScope: Pixel-Grounded Multimodal Understanding for Agricultural Imagescited by 0
  7. 2026Toward Interactive Understanding of Code APIscited by 0