# Reassessing Claims of Human Parity and Super-Human Performance in Machine Translation at WMT 2019

arXiv: https://arxiv.org/abs/2005.05738 · HTML: https://mlsanity.com/paper/2005.05738/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 May 2020.

## Built on by (most cited first, 20 of 20)

- [Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation](https://mlsanity.com/paper/2104.14478/) (2021)
- [High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metrics](https://mlsanity.com/paper/2111.09388/) (2021)
- [Spinning Language Models: Risks of Propaganda-As-A-Service and Countermeasures](https://mlsanity.com/paper/2112.05224/) (2021)
- [Embarrassingly Easy Document-Level MT Metrics: How to Convert Any Pretrained Metric Into a Document-Level Metric](https://mlsanity.com/paper/2209.13654/) (2022)
- [Uncertainty-Aware Machine Translation Evaluation](https://mlsanity.com/paper/2109.06352/) (2021)
- [It Takes Two to Tango: Navigating Conceptualizations of NLP Tasks and Measurements of Performance](https://mlsanity.com/paper/2305.09022/) (2023)
- [Fine-Tuned Machine Translation Metrics Struggle in Unseen Domains](https://mlsanity.com/paper/2402.18747/) (2024)
- [Toward Human-Like Evaluation for Natural Language Generation with Error Analysis](https://mlsanity.com/paper/2212.10179/) (2022)
- [Multilingual Contextualization of Large Language Models for Document-Level Machine Translation](https://mlsanity.com/paper/2504.12140/) (2025)
- [Disentangling Uncertainty in Machine Translation Evaluation](https://mlsanity.com/paper/2204.06546/) (2022)
- [Assessing Reference-Free Peer Evaluation for Machine Translation](https://mlsanity.com/paper/2104.05146/) (2021)
- [A Human Subject Study of Named Entity Recognition (NER) in Conversational Music Recommendation Queries](https://mlsanity.com/paper/2303.06944/) (2023)
- [Spinning Sequence-to-Sequence Models with Meta-Backdoors](https://mlsanity.com/paper/2107.10443/) (2021)
- [Automatic Classification of Human Translation and Machine Translation: A Study from the Perspective of Lexical Diversity](https://mlsanity.com/paper/2105.04616/) (2021)
- [On Systematic Style Differences between Unsupervised and Supervised MT and an Application for High-Resource Machine Translation](https://mlsanity.com/paper/2106.15818/) (2021)
- [On User Interfaces for Large-Scale Document-Level Human Evaluation of Machine Translation Outputs](https://mlsanity.com/paper/2104.10408/) (2021)
- [The Comparison of Translationese in Machine Translation and Human Transation in terms of Translation Relations](https://mlsanity.com/paper/2404.08661/) (2024)
- [Segment-Based Interactive Machine Translation for Pre-trained Models](https://mlsanity.com/paper/2407.06990/) (2024)
- [AI Technologies in Language Access: Attitudes Towards AI and the Human Value of Language Access Managers](https://mlsanity.com/paper/2605.19234/) (2026)
- [The Blindness of Document-Level Translation Evaluation](https://mlsanity.com/paper/2609.05949/) (2026)

## Abstract (arXiv)

We reassess the claims of human parity and super-human performance made at the news shared task of WMT 2019 for three translation directions: English-to-German, English-to-Russian and German-to-English. First we identify three potential issues in the human evaluation of that shared task: (i) the limited amount of intersentential context available, (ii) the limited translation proficiency of the evaluators and (iii) the use of a reference translation. We then conduct a modified evaluation taking these issues into account. Our results indicate that all the claims of human parity and super-human performance made at WMT 2019 should be refuted, except the claim of human parity for English-to-German. Based on our findings, we put forward a set of recommendations and open questions for future assessments of human parity in machine translation.