mlsanityData to 7 Oct 2026

What Are We Measuring in NLG? A Meta-Analysis of Evaluation Trends 2020-2025

arXiv 2601.07648 ↗First posted 12 Jan 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

As Natural Language Generation (NLG) dominates modern NLP, scalable evaluation remains a critical bottleneck. Consequently, LLM-as-a-judge (LaaJ) adoption has accelerated rapidly, appearing in more papers than human evaluation in 2025. This pivotal shift motivates a critical analysis of current evaluation practices. Overcoming the limits of rigid keyword filtering and manual review, we employ a multi-LLM information extraction pipeline to gather structured metadata from 14,171 papers across four major NLP conferences (2020-2025). Analyzing 3,334 filtered NLG papers, we identify three systemic challenges. (1) Metric inertia: despite the shift toward open-ended generation, legacy lexical metrics (BLEU, ROUGE) persist as primary indicators, typically used alongside rather than replaced by semantic alternatives. (2) Metric-criteria mapping problem: our paper-level co-occurrence data reveals that general-purpose automatic metrics are applied as broad proxies for quality, without specifying which dimension of text generation they are intended to evaluate. (3) Validation gap: LaaJ has grown rapidly without commensurate human validation (fewer than 8% of papers). Crucially, while LaaJ correlates with aggregate quality, alignment collapses on fine-grained criteria like fluency. To address these gaps, we distill our findings into a minimal Evaluation Checklist to guide metric selection, construct validity, and LaaJ deployment.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 29 of its 45 references that are in the map. First posted on arXiv 12 Jan 2026.

Builds on

29 of 45 references.

References · 29

  1. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  2. 2024A Survey on LLM-as-a-Judgecited by 1,384
  3. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  4. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  5. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  6. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  7. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  8. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  9. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 415
  10. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 402
  11. 2023AlignScore: Evaluating Factual Consistency with a Unified Alignment Functioncited by 284
  12. 2020Evaluation of Text Generation: A Surveycited by 275
17 more
  1. 2024MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documentscited by 251
  2. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 236
  3. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 233
  4. 2024Large Language Models for Data Annotation and Synthesis: A Surveycited by 201
  5. 2020A Survey of Evaluation Metrics Used for NLG Systemscited by 164
  6. 2020Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metricscited by 155
  7. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  8. 2024An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4cited by 90
  9. 2024LLMs as Research Tools: A Large Scale Survey of Researchers' Usage and Perceptionscited by 45
  10. 2024Are LLM-based Evaluators Confusing NLG Quality Criteria?cited by 35
  11. 2024Evaluating the Consistency of LLM Evaluatorscited by 24
  12. 2024The emergence of Large Language Models (LLM) as a tool in literature reviews: an LLM automated systematic reviewcited by 20
  13. 2024Automatic Metrics in Natural Language Generation: A Survey of Current Evaluation Practicescited by 19
  14. 2024DHP Benchmark: Are LLMs Good NLG Evaluators?cited by 12
  15. 2025MetricX-25 and GemSpanEval: Google Translate Submissions to the WMT25 Evaluation Shared Taskcited by 10
  16. 2021Language Model Augmented Relevance Scorecited by 7
  17. 2025MOLE: Metadata Extraction and Validation in Scientific Papers Using LLMscited by 4

Built on by

Not cited yet.