mlsanityData to 7 Oct 2026

Evaluating Evaluation Metrics: A Framework for Analyzing NLG Evaluation Metrics using Measurement Theory

arXiv 2305.14889 ↗First posted 24 May 2023cs.CL cs.AIcited by 41

Not on the map. Why?

Abstract (arXiv)

We address a fundamental challenge in Natural Language Generation (NLG) model evaluation -- the design and evaluation of evaluation metrics. Recognizing the limitations of existing automatic metrics and noises from how current human evaluation was conducted, we propose MetricEval, a framework informed by measurement theory, the foundation of educational test design, for conceptualizing and evaluating the reliability and validity of NLG evaluation metrics. The framework formalizes the source of measurement error and offers statistical tools for evaluating evaluation metrics based on empirical data. With our framework, one can quantify the uncertainty of the metrics to better interpret the result. To exemplify the use of our framework in practice, we analyzed a set of evaluation metrics for summarization and identified issues related to conflated validity structure in human-eval and reliability in LLM-based metrics. Through MetricEval, we aim to promote the design, evaluation, and interpretation of valid and reliable metrics to advance robust and effective NLG models.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Builds on 33 of its 63 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

33 of 63 references.

References · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  4. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  5. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  6. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  7. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  8. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  9. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 415
  10. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
  11. 2018Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategiescited by 292
  12. 2021All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Textcited by 287
21 more
  1. 2020Evaluation of Text Generation: A Surveycited by 275
  2. 2017Why We Need New Evaluation Metrics for NLGcited by 232
  3. 2021Societal Biases in Language Generation: Progress and Challengescited by 201
  4. 2020A Survey of Evaluation Metrics Used for NLG Systemscited by 164
  5. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  6. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 141
  7. 2019Answers Unite! Unsupervised Metrics for Reinforced Summarization Modelscited by 92
  8. 2017RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systemscited by 88
  9. 2020Fill in the BLANC: Human-free quality estimation of document summariescited by 80
  10. 2021A Statistical Analysis of Summarization Evaluation Metrics using Resampling Methodscited by 71
  11. 2020SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarizationcited by 69
  12. 2015Better Summarization Evaluation with Word Embeddings for ROUGEcited by 66
  13. 2018Towards a Better Metric for Evaluating Question Generation Systemscited by 52
  14. 2021Perturbation CheckLists for Evaluating NLG Evaluation Metricscited by 49
  15. 2023SEAHORSE: A Multilingual, Multifaceted Dataset for Summarization Evaluationcited by 42
  16. 2022Deconstructing NLG Evaluation: Evaluation Practices, Assumptions, and Their Implicationscited by 32
  17. 2020Curious Case of Language Generation Evaluation Metrics: A Cautionary Talecited by 28
  18. 2021GENIE: Toward Reproducible and Standardized Human Evaluation for Text Generationcited by 20
  19. 2020Defining and Evaluating Fair Natural Language Generationcited by 17
  20. 2023NLG Evaluation Metrics Beyond Correlation Analysis: An Empirical Metric Preference Checklistcited by 16
  21. 2022On the Effectiveness of Automated Metrics for Text Generation Systemscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 15; 2025: 14; 2026: 9.

41 papers.

Sort:

By area:

  1. 2024CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklistscited by 87
  2. 2024LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Textscited by 81
  3. 2024LLM-based NLG Evaluation: Current Status and Challengescited by 71
  4. 2023Value FULCRA: Mapping Large Language Models to the Multidimensional Spectrum of Basic Human Valuescited by 49
  5. 2025Neither Valid nor Reliable? Investigating the Use of LLMs as Judgescited by 37
  6. 2025Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworkscited by 37
  7. 2024Position: Measure Dataset Diversity, Don't Just Claim Itcited by 26
  8. 2024What Can Natural Language Processing Do for Peer Review?cited by 24
  9. 2025SePer: Measure Retrieval Utility Through The Lens Of Semantic Perplexity Reductioncited by 24
  10. 2023Evaluating General-Purpose AI with Psychometricscited by 23
  11. 2024Evaluating Large Language Models with Psychometricscited by 22
  12. 2024Raising the Bar: Investigating the Values of Large Language Models via Generative Evolving Testingcited by 17
29 more
  1. 2024SLIDE: A Framework Integrating Small and Large Language Models for Open-Domain Dialogues Evaluationcited by 11
  2. 2024What makes a good metric? Evaluating automatic metrics for text-to-image consistencycited by 11
  3. 2024Evaluating Readability and Faithfulness of Concept-based Explanationscited by 10
  4. 2023Evaluating Task-oriented Dialogue Systems: A Systematic Review of Measures, Constructs and their Operationalisationscited by 9
  5. 2024Are Small Language Models Ready to Compete with Large Language Models for Practical Applications?cited by 9
  6. 2024Large Language Models Are Active Critics in NLG Evaluationcited by 6
  7. 2025Value Compass Benchmarks: A Platform for Fundamental and Validated Evaluation of LLMs Valuescited by 6
  8. 2025Quantifying Language Disparities in Multilingual Large Language Modelscited by 6
  9. 2025Measuring What Matters: Connecting AI Ethics Evaluations to System Attributes, Hazards, and Harmscited by 5
  10. 2025SimBA: Simplifying Benchmark Analysis Using Performance Matrices Alonecited by 5
  11. 2024Exploring Bengali Religious Dialect Biases in Large Language Models with Evaluation Perspectivescited by 4
  12. 2025AdAEM: An Adaptively and Automated Extensible Measurement of LLMs' Value Differencecited by 4
  13. 2025Specification-Aware Machine Translation and Evaluation for Purpose Alignmentcited by 4
  14. 2025Understanding and Meeting Practitioner Needs When Measuring Representational Harms Caused by LLM-Based Systemscited by 3
  15. 2025Contextual Metric Meta-Evaluation by Measuring Local Metric Accuracycited by 2
  16. 2025MEQA: A Meta-Evaluation Framework for Question & Answer LLM Benchmarkscited by 2
  17. 2025A Qualitative Investigation into LLM-Generated Multilingual Code Comments and Automatic Evaluation Metricscited by 2
  18. 2026The Double Measurement Confound in Agent Benchmarks: De-Scaffolding, Ground-Truth Scoring, and Reliability Beyond the Meancited by 2
  19. 2024Reliability of Topic Modelingcited by 1
  20. 2025DRE: An Effective Dual-Refined Method for Integrating Small and Large Language Models in Open-Domain Dialogue Evaluationcited by 1
  21. 2026AI Evaluation Should Require Standardized Item-Level Data Releasescited by 1
  22. 2026Distributional Open-Ended Evaluation of LLM Cultural Value Alignment Based on Value Codebookcited by 1
  23. 2026FairTree: Subgroup Fairness Auditing of Machine Learning Models with Bias-Variance Decompositioncited by 1
  24. 2026Do Assessment Instruments Measure the Same Thing for Humans and LLMs? A Latent Structure Analysiscited by 1
  25. 2024A Novel Psychometrics-Based Approach to Developing Professional Competency Benchmark for Large Language Modelscited by 0
  26. 2026Results-Actionability Gap: Understanding How Practitioners Evaluate LLM Products in the Wildcited by 0
  27. 2026Accuracy Hides How Language Models Fail: Measuring Failure States Under Matched Output Budgetscited by 0
  28. 2026Frontier AI Forecasting Has a Measurement Problem: An Audit of Progress Evidencecited by 0
  29. 2026LLJ Cards: Best practices for the Use of LLMs as Judgescited by 0