mlsanityData to 14 Oct 2026

On the Effectiveness of Automated Metrics for Text Generation Systems

arXiv 2210.13025 ↗First posted 24 Oct 2022cs.CL cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

A major challenge in the field of Text Generation is evaluation because we lack a sound theory that can be leveraged to extract guidelines for evaluation campaigns. In this work, we propose a first step towards such a theory that incorporates different sources of uncertainty, such as imperfect automated metrics and insufficiently sized test sets. The theory has practical applications, such as determining the number of samples needed to reliably distinguish the performance of a set of Text Generation systems in a given setting. We showcase the application of the theory on the WMT 21 and Spot-The-Bot evaluation data and outline how it can be leveraged to improve the evaluation protocol regarding the reliability, robustness, and significance of the evaluation outcome.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 21 of its 38 references that are in the map. First posted on arXiv 24 Oct 2022.

Builds on

21 of 38 references.

References · 21

  1. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  2. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  3. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  4. 2020Recipes for building an open-domain chatbotcited by 593
  5. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 338
  6. 2021All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Textcited by 287
  7. 2020Evaluation of Text Generation: A Surveycited by 277
  8. 2017Why We Need New Evaluation Metrics for NLGcited by 233
  9. 2020USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generationcited by 188
  10. 2019The Second Conversational Intelligence Challenge (ConvAI2)cited by 179
  11. 2020A Survey of Evaluation Metrics Used for NLG Systemscited by 164
  12. 2020With Little Power Comes Great Responsibilitycited by 142
9 more
  1. 2019Unifying Human and Statistical Evaluation for Natural Language Generationcited by 127
  2. 2019Survey on Evaluation Methods for Dialogue Systemscited by 126
  3. 2018RankME: Reliable Human Ratings for Natural Language Generationcited by 69
  4. 2018The price of debiasing automatic metrics in natural language evaluationcited by 60
  5. 2021Facebook AI WMT21 News Translation Task Submissioncited by 57
  6. 2020Spot The Bot: A Robust and Efficient Framework for the Evaluation of Conversational Dialogue Systemscited by 26
  7. 2019Towards Best Experiment Design for Evaluating Dialogue System Outputcited by 15
  8. 2021The Volctrans GLAT System: Non-autoregressive Translation Meets WMT21cited by 14
  9. 2022Active Evaluation: Efficient NLG Evaluation with Few Pairwise Comparisonscited by 10

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 0; 2025: 1.

4 papers.

  1. 2023Evaluating Evaluation Metrics: A Framework for Analyzing NLG Evaluation Metrics using Measurement Theorycited by 43
  2. 2023Text Style Transfer Evaluation Using Large Language Modelscited by 13
  3. 2023Correction of Errors in Preference Ratings from Automated Metrics for Text Generationcited by 3
  4. 2025LeCoDe: A Benchmark Dataset for Interactive Legal Consultation Dialogue Evaluationcited by 2