mlsanityData to 7 Oct 2026

Reassessing Claims of Human Parity and Super-Human Performance in Machine Translation at WMT 2019

arXiv 2005.05738 ↗First posted 12 May 2020cs.CLcited by 20

Not on the map. Why?

Abstract (arXiv)

We reassess the claims of human parity and super-human performance made at the news shared task of WMT 2019 for three translation directions: English-to-German, English-to-Russian and German-to-English. First we identify three potential issues in the human evaluation of that shared task: (i) the limited amount of intersentential context available, (ii) the limited translation proficiency of the evaluators and (iii) the use of a reference translation. We then conduct a modified evaluation taking these issues into account. Our results indicate that all the claims of human parity and super-human performance made at WMT 2019 should be refuted, except the claim of human parity for English-to-German. Based on our findings, we put forward a set of recommendations and open questions for future assessments of human parity in machine translation.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). First posted on arXiv 12 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 3; 2023: 2; 2024: 3; 2025: 1; 2026: 2.

20 papers.

Sort:

By area:

  1. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 333
  2. 2021High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metricscited by 104
  3. 2021Spinning Language Models: Risks of Propaganda-As-A-Service and Countermeasurescited by 69
  4. 2022Embarrassingly Easy Document-Level MT Metrics: How to Convert Any Pretrained Metric Into a Document-Level Metriccited by 56
  5. 2021Uncertainty-Aware Machine Translation Evaluationcited by 35
  6. 2023It Takes Two to Tango: Navigating Conceptualizations of NLP Tasks and Measurements of Performancecited by 25
  7. 2024Fine-Tuned Machine Translation Metrics Struggle in Unseen Domainscited by 21
  8. 2022Toward Human-Like Evaluation for Natural Language Generation with Error Analysiscited by 18
  9. 2025Multilingual Contextualization of Large Language Models for Document-Level Machine Translationcited by 11
  10. 2022Disentangling Uncertainty in Machine Translation Evaluationcited by 9
  11. 2021Assessing Reference-Free Peer Evaluation for Machine Translationcited by 8
  12. 2023A Human Subject Study of Named Entity Recognition (NER) in Conversational Music Recommendation Queriescited by 6
8 more
  1. 2021Spinning Sequence-to-Sequence Models with Meta-Backdoorscited by 5
  2. 2021Automatic Classification of Human Translation and Machine Translation: A Study from the Perspective of Lexical Diversitycited by 4
  3. 2021On Systematic Style Differences between Unsupervised and Supervised MT and an Application for High-Resource Machine Translationcited by 4
  4. 2021On User Interfaces for Large-Scale Document-Level Human Evaluation of Machine Translation Outputscited by 3
  5. 2024The Comparison of Translationese in Machine Translation and Human Transation in terms of Translation Relationscited by 1
  6. 2024Segment-Based Interactive Machine Translation for Pre-trained Modelscited by 0
  7. 2026AI Technologies in Language Access: Attitudes Towards AI and the Human Value of Language Access Managerscited by 0
  8. 2026The Blindness of Document-Level Translation Evaluationcited by 0