mlsanityData to 7 Oct 2026

Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation

arXiv 2104.14478 ↗First posted 29 Apr 2021cs.CL cs.AI cs.LGcited by 333

Not on the map. Why?

Abstract (arXiv)

Human evaluation of modern high-quality machine translation systems is a difficult problem, and there is increasing evidence that inadequate evaluation procedures can lead to erroneous conclusions. While there has been considerable research on human evaluation, the field still lacks a commonly-accepted standard procedure. As a step toward this goal, we propose an evaluation methodology grounded in explicit error analysis, based on the Multidimensional Quality Metrics (MQM) framework. We carry out the largest MQM research study to date, scoring the outputs of top systems from the WMT 2020 shared task in two language pairs using annotations provided by professional translators with access to full document context. We analyze the resulting data extensively, finding among other results a substantially different ranking of evaluated systems from the one established by the WMT crowd workers, exhibiting a clear preference for human over machine output. Surprisingly, we also find that automatic metrics based on pre-trained embeddings can outperform human crowd workers. We make our corpus publicly available for further research.

Key facts. Cited by 333 papers in the map (data through 7 Oct 2026). Builds on 13 of its 36 references that are in the map. First posted on arXiv 29 Apr 2021.

Builds on

13 of 36 references.

References · 13

  1. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  2. 2018Achieving Human Parity on Automatic Chinese to English News Translationcited by 119
  3. 2020BLEU might be Guilty but References are not Innocentcited by 109
  4. 2018Has Machine Translation Achieved Human Parity? A Case for Document-level Evaluationcited by 88
  5. 2018Attaining the Unattainable? Reassessing Claims of Human Parity in Neural Machine Translationcited by 58
  6. 2019The Effect of Translationese in Machine Translation Test Setscited by 49
  7. 2019Translationese in Machine Translation Evaluationcited by 38
  8. 2020A Set of Recommendations for Assessing Human-Machine Parity in Language Translationcited by 34
  9. 2020A Gold Standard Methodology for Evaluating Accuracy in Data-To-Text Systemscited by 34
  10. 2020Reassessing Claims of Human Parity and Super-Human Performance in Machine Translation at WMT 2019cited by 20
  11. 2018Quantitative Fine-Grained Human Evaluation of Machine Translation Systems: a Case Study on English to Croatiancited by 7
  12. 2020What's the Difference Between Professional Human and Machine Translation? A Blind Multi-language Study on Domain-specific MTcited by 5
1 more
  1. 2018Exploring Gap Filling as a Cheaper Alternative to Reading Comprehension Questionnaires when Evaluating Machine Translation for Gistingcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 25; 2022: 54; 2023: 51; 2024: 74; 2025: 69; 2026: 60.

333 papers.

Sort:

By area:

  1. 2023PaLM 2 Technical Reportcited by 1,154
  2. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 416
  3. 2023GPTScore: Evaluate as You Desirecited by 357
  4. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 329
  5. 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 260
  6. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 236
  7. 2022CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Taskcited by 205
  8. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 172
  9. 2021To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translationcited by 161
  10. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  11. 2022Prompting PaLM for Translation: Assessing Strategies and Performancecited by 144
  12. 2023GEMBA-MQM: Detecting Translation Quality Error Spans with GPT-4cited by 143
38 more
  1. 2023BooookScore: A systematic exploration of book-length summarization in the era of LLMscited by 137
  2. 2023A Critical Evaluation of Evaluations for Long-form Question Answeringcited by 111
  3. 2021High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metricscited by 104
  4. 2023The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluationcited by 95
  5. 2021Learning Compact Metrics for MTcited by 93
  6. 2021Survey of Low-Resource Machine Translationcited by 92
  7. 2024MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Taskcited by 92
  8. 2023The unreasonable effectiveness of few-shot learning for machine translationcited by 91
  9. 2022On Decoding Strategies for Neural Text Generatorscited by 89
  10. 2023Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Modelscited by 87
  11. 2023Large language models effectively leverage document-level context for literary translation, but critical errors persistcited by 80
  12. 2023Bactrian-X: Multilingual Replicable Instruction-Following Models with Low-Rank Adaptationcited by 79
  13. 2021Evaluating Attribution in Dialogue Systems: The BEGIN Benchmarkcited by 75
  14. 2023AutomaTikZ: Text-Guided Synthesis of Scientific Vector Graphics with TikZcited by 74
  15. 2024Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluationcited by 72
  16. 2023Ties Matter: Meta-Evaluating Modern Metrics with Pairwise Accuracy and Tie Calibrationcited by 71
  17. 2022Building Machine Translation Systems for the Next Thousand Languagescited by 69
  18. 2023TIGERScore: Towards Building Explainable Metric for All Text Generation Taskscited by 69
  19. 2024Error Span Annotation: A Balanced Approach for Human Evaluation of Machine Translationcited by 62
  20. 2023Epsilon Sampling Rocks: Investigating Sampling Strategies for Minimum Bayes Risk Decoding for Machine Translationcited by 57
  21. 2022Embarrassingly Easy Document-Level MT Metrics: How to Convert Any Pretrained Metric Into a Document-Level Metriccited by 56
  22. 2023Leveraging GPT-4 for Automatic Translation Post-Editingcited by 56
  23. 2022Data Scaling Laws in NMT: The Effect of Noise and Architecturecited by 55
  24. 2023Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generationcited by 55
  25. 2021Perturbation CheckLists for Evaluating NLG Evaluation Metricscited by 49
  26. 2022Quality-Aware Decoding for Neural Machine Translationcited by 49
  27. 2024(Perhaps) Beyond Human Translation: Harnessing Multi-Agent Collaboration for Translating Ultra-Long Literary Textscited by 49
  28. 2025MT-R1-Zero: Advancing LLM-based Machine Translation via R1-Zero-like Reinforcement Learningcited by 48
  29. 2023Exploring Human-Like Translation Strategy with Large Language Modelscited by 44
  30. 2023SEAHORSE: A Multilingual, Multifaceted Dataset for Summarization Evaluationcited by 42
  31. 2022Identifying Weaknesses in Machine Translation Metrics Through Minimum Bayes Risk Decoding: A Case Study for COMETcited by 41
  32. 2021Transparent Human Evaluation for Image Captioningcited by 40
  33. 2024Navigating the Metrics Maze: Reconciling Score Magnitudes and Accuraciescited by 40
  34. 2023LLMRefine: Pinpointing and Refining Large Language Models via Fine-Grained Actionable Feedbackcited by 39
  35. 2022MT-GenEval: A Counterfactual and Contextual Dataset for Evaluating Gender Accuracy in Machine Translationcited by 38
  36. 2024Leveraging Large Language Models for NLG Evaluation: Advances and Challengescited by 37
  37. 2024TEaR: Improving LLM-based Machine Translation with Systematic Self-Refinementcited by 37
  38. 2021Uncertainty-Aware Machine Translation Evaluationcited by 35

50 most cited of 333 All 333 (JSON)