mlsanityData to 7 Oct 2026

Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics

arXiv 2006.06264 ↗First posted 11 Jun 2020cs.CLcited by 155

Not on the map. Why?

Abstract (arXiv)

Automatic metrics are fundamental for the development and evaluation of machine translation systems. Judging whether, and to what extent, automatic metrics concur with the gold standard of human evaluation is not a straightforward problem. We show that current methods for judging metrics are highly sensitive to the translations used for assessment, particularly the presence of outliers, which often leads to falsely confident conclusions about a metric's efficacy. Finally, we turn to pairwise system ranking, developing a method for thresholding performance improvement under an automatic metric against human judgements, which allows quantification of type I versus type II errors incurred, i.e., insignificant human differences in system quality that are accepted, and significant human differences that are rejected. Together, these findings suggest improvements to the protocols for metric evaluation and system performance evaluation in machine translation.

Key facts. Cited by 155 papers in the map (data through 7 Oct 2026). First posted on arXiv 11 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 31; 2022: 32; 2023: 22; 2024: 30; 2025: 16; 2026: 24.

155 papers.

Sort:

By area:

  1. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  2. 2021The GEM Benchmark: Natural Language Generation, its Evaluation and Metricscited by 239
  3. 2021To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translationcited by 161
  4. 2023GEMBA-MQM: Detecting Translation Quality Error Spans with GPT-4cited by 143
  5. 2022Pretrained Language Models for Text Generation: A Surveycited by 121
  6. 2021Learning Compact Metrics for MTcited by 93
  7. 2023LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scorescited by 88
  8. 2023Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Modelscited by 87
  9. 2021Evaluating Attribution in Dialogue Systems: The BEGIN Benchmarkcited by 75
  10. 2021What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformerscited by 75
  11. 2022Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generationcited by 73
  12. 2023Ties Matter: Meta-Evaluating Modern Metrics with Pairwise Accuracy and Tie Calibrationcited by 71
38 more
  1. 2023Efficient Benchmarking of Language Modelscited by 71
  2. 2021Scientific Credibility of Machine Translation Research: A Meta-Evaluation of 769 Paperscited by 70
  3. 2022Out of the BLEU: how should we assess quality of the Code Generation models?cited by 65
  4. 2021OpenMEVA: A Benchmark for Evaluating Open-ended Story Generation Metricscited by 61
  5. 2021Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generationcited by 60
  6. 2024Exploring LLM-based Agents for Root Cause Analysiscited by 60
  7. 2023Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generationcited by 55
  8. 2021Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarkingcited by 53
  9. 2021Perturbation CheckLists for Evaluating NLG Evaluation Metricscited by 49
  10. 2022Quality-Aware Decoding for Neural Machine Translationcited by 49
  11. 2021Shades of BLEU, Flavours of Success: The Case of MultiWOZcited by 48
  12. 2022On the Blind Spots of Model-Based Evaluation Metrics for Text Generationcited by 48
  13. 2024Navigating the Metrics Maze: Reconciling Score Magnitudes and Accuraciescited by 40
  14. 2022Re-Examining System-Level Correlations of Automatic Summarization Evaluation Metricscited by 37
  15. 2022DEMETR: Diagnosing Evaluation Metrics for Translationcited by 36
  16. 2022Calibrated Interpretation: Confidence Estimation in Semantic Parsingcited by 33
  17. 2021Bidimensional Leaderboards: Generate and Evaluate Language Hand in Handcited by 32
  18. 2024Finding Blind Spots in Evaluator LLMs with Interpretable Checklistscited by 32
  19. 2021Better than Average: Paired Evaluation of NLP Systemscited by 30
  20. 2022UniTE: Unified Translation Evaluationcited by 28
  21. 2022IndicMT Eval: A Dataset to Meta-Evaluate Machine Translation metrics for Indian Languagescited by 27
  22. 2022Spurious Correlations in Reference-Free Evaluation of Text Generationcited by 26
  23. 2021Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factorscited by 25
  24. 2024Improving Statistical Significance in Human Evaluation of Automatic Metrics via Soft Pairwise Accuracycited by 25
  25. 2021How Robust are Model Rankings: A Leaderboard Customization Approach for Equitable Evaluationcited by 24
  26. 2023BLEURT Has Universal Translations: An Analysis of Automatic Metrics by Minimum Risk Trainingcited by 24
  27. 2021Control Prefixes for Parameter-Efficient Text Generationcited by 23
  28. 2023Exploring the Reliability of Large Language Models as Customized Evaluators for Diverse NLP Taskscited by 23
  29. 2021BLEU, METEOR, BERTScore: Evaluation of Metrics Performance in Assessing Critical Translation Errors in Sentiment-oriented Textcited by 22
  30. 2024The Fine-Tuning Paradox: Boosting Translation Quality Without Sacrificing LLM Abilitiescited by 22
  31. 2024MQM-APE: Toward High-Quality Error Annotation Predictors with Automatic Post-Editing in LLM Translation Evaluatorscited by 22
  32. 2021Machine Translation into Low-resource Language Varietiescited by 19
  33. 2021The statistical advantage of automatic NLG metrics at the system levelcited by 18
  34. 2022MENLI: Robust Evaluation Metrics from Natural Language Inferencecited by 18
  35. 2022T5Score: Discriminative Fine-tuning of Generative Evaluation Metricscited by 18
  36. 2022Toward Human-Like Evaluation for Natural Language Generation with Error Analysiscited by 18
  37. 2023Which Prompts Make The Difference? Data Prioritization For Efficient Human LLM Evaluationcited by 18
  38. 2025BLEUBERI: BLEU is a surprisingly effective reward for instruction followingcited by 17

50 most cited of 155 All 155 (JSON)