mlsanityData to 7 Oct 2026

To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation

arXiv 2107.10821 ↗First posted 22 Jul 2021cs.CLcited by 161

Not on the map. Why?

Abstract (arXiv)

Automatic metrics are commonly used as the exclusive tool for declaring the superiority of one machine translation system's quality over another. The community choice of automatic metric guides research directions and industrial developments by deciding which models are deemed better. Evaluating metrics correlations with sets of human judgements has been limited by the size of these sets. In this paper, we corroborate how reliable metrics are in contrast to human judgements on -- to the best of our knowledge -- the largest collection of judgements reported in the literature. Arguably, pairwise rankings of two systems are the most common evaluation tasks in research or deployment scenarios. Taking human judgement as a gold standard, we investigate which metrics have the highest accuracy in predicting translation quality rankings for such system pairs. Furthermore, we evaluate the performance of various metrics across different language pairs and domains. Lastly, we show that the sole use of BLEU impeded the development of improved models leading to bad deployment decisions. We release the collection of 2.3M sentence-level human judgements for 4380 systems for further analysis and replication of our work.

Key facts. Cited by 161 papers in the map (data through 7 Oct 2026). Builds on 14 of its 47 references that are in the map. First posted on arXiv 22 Jul 2021.

Builds on

14 of 47 references.

References · 14

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  4. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  5. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  6. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  7. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 333
  8. 2020Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metricscited by 155
  9. 2020Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasingcited by 114
  10. 2020BLEU might be Guilty but References are not Innocentcited by 109
  11. 2021Scientific Credibility of Machine Translation Research: A Meta-Evaluation of 769 Paperscited by 70
  12. 2019The Effect of Translationese in Machine Translation Test Setscited by 49
2 more
  1. 2019Fluent Translations from Disfluent Speech in End-to-End Speech Translationcited by 22
  2. 2021Assessing Reference-Free Peer Evaluation for Machine Translationcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 37; 2023: 41; 2024: 41; 2025: 22; 2026: 15.

161 papers.

Sort:

By area:

  1. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 416
  2. 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 260
  3. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  4. 2023Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Modelscited by 152
  5. 2022Prompting PaLM for Translation: Assessing Strategies and Performancecited by 144
  6. 2023GEMBA-MQM: Detecting Translation Quality Error Spans with GPT-4cited by 143
  7. 2023The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluationcited by 95
  8. 2021Survey of Low-Resource Machine Translationcited by 92
  9. 2024MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Taskcited by 92
  10. 2023The unreasonable effectiveness of few-shot learning for machine translationcited by 91
  11. 2023Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Modelscited by 87
  12. 2022Detecting and Mitigating Hallucinations in Machine Translation: Model Internal Workings Alone Do Well, Sentence Similarity Even Bettercited by 75
38 more
  1. 2023Ties Matter: Meta-Evaluating Modern Metrics with Pairwise Accuracy and Tie Calibrationcited by 71
  2. 2023Efficient Benchmarking of Language Modelscited by 71
  3. 2022Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluationcited by 66
  4. 2024Error Span Annotation: A Balanced Approach for Human Evaluation of Machine Translationcited by 62
  5. 2022Looking for a Needle in a Haystack: A Comprehensive Study of Hallucinations in Neural Machine Translationcited by 60
  6. 2022Embarrassingly Easy Document-Level MT Metrics: How to Convert Any Pretrained Metric Into a Document-Level Metriccited by 56
  7. 2022Data Scaling Laws in NMT: The Effect of Noise and Architecturecited by 55
  8. 2022LENS: A Learnable Evaluation Metric for Text Simplificationcited by 54
  9. 2024Quantifying Variance in Evaluation Benchmarkscited by 52
  10. 2022GODEL: Large-Scale Pre-Training for Goal-Directed Dialogcited by 50
  11. 2023ICE-Score: Instructing Large Language Models to Evaluate Codecited by 50
  12. 2022Quality-Aware Decoding for Neural Machine Translationcited by 49
  13. 2024LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, A Technical Reportcited by 47
  14. 2023Exploring Human-Like Translation Strategy with Large Language Modelscited by 44
  15. 2023SEAHORSE: A Multilingual, Multifaceted Dataset for Summarization Evaluationcited by 42
  16. 2022Identifying Weaknesses in Machine Translation Metrics Through Minimum Bayes Risk Decoding: A Case Study for COMETcited by 41
  17. 2024Navigating the Metrics Maze: Reconciling Score Magnitudes and Accuraciescited by 40
  18. 2024TEaR: Improving LLM-based Machine Translation with Systematic Self-Refinementcited by 37
  19. 2022DEMETR: Diagnosing Evaluation Metrics for Translationcited by 36
  20. 2022DiscoScore: Evaluating Text Generation with BERT and Discourse Coherencecited by 35
  21. 2023SLTUNET: A Simple Unified Model for Sign Language Translationcited by 35
  22. 2024Improving Machine Translation with Human Feedback: An Exploration of Quality Estimation as a Reward Modelcited by 33
  23. 2023Benchmarking Large Language Model Capabilities for Conditional Generationcited by 29
  24. 2022UniTE: Unified Translation Evaluationcited by 28
  25. 2023Speech Translation with Large Language Models: An Industrial Practicecited by 28
  26. 2022IndicMT Eval: A Dataset to Meta-Evaluate Machine Translation metrics for Indian Languagescited by 27
  27. 2022Spurious Correlations in Reference-Free Evaluation of Text Generationcited by 26
  28. 2024Open-ended VQA benchmarking of Vision-Language models by exploiting Classification datasets and their semantic hierarchycited by 26
  29. 2022ACES: Translation Accuracy Challenge Sets for Evaluating Machine Translation Metricscited by 25
  30. 2024Improving Statistical Significance in Human Evaluation of Automatic Metrics via Soft Pairwise Accuracycited by 25
  31. 2022Considerations for meaningful sign language machine translation based on glossescited by 24
  32. 2023BLEURT Has Universal Translations: An Analysis of Automatic Metrics by Minimum Risk Trainingcited by 24
  33. 2024NADI 2024: The Fifth Nuanced Arabic Dialect Identification Shared Taskcited by 23
  34. 2023What's the Meaning of Superhuman Performance in Today's NLU?cited by 22
  35. 2024The Fine-Tuning Paradox: Boosting Translation Quality Without Sacrificing LLM Abilitiescited by 22
  36. 2024MQM-APE: Toward High-Quality Error Annotation Predictors with Automatic Post-Editing in LLM Translation Evaluatorscited by 22
  37. 2024Guardians of the Machine Translation Meta-Evaluation: Sentinel Metrics Fall In!cited by 21
  38. 2023JWSign: A Highly Multilingual Corpus of Bible Translations for more Diversity in Sign Language Processingcited by 18

50 most cited of 161 All 161 (JSON)