# A Set of Recommendations for Assessing Human-Machine Parity in Language Translation

arXiv: https://arxiv.org/abs/2004.01694 · HTML: https://mlsanity.com/paper/2004.01694/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Apr 2020.

## Built on by (most cited first, 30 of 34)

- [Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation](https://mlsanity.com/paper/2104.14478/) (2021)
- [Scientific Credibility of Machine Translation Research: A Meta-Evaluation of 769 Papers](https://mlsanity.com/paper/2106.15195/) (2021)
- [BlonDe: An Automatic Evaluation Metric for Document-level Machine Translation](https://mlsanity.com/paper/2103.11878/) (2021)
- [Translation Quality Assessment: A Brief Survey on Manual and Automatic Methods](https://mlsanity.com/paper/2105.03311/) (2021)
- [What's the Meaning of Superhuman Performance in Today's NLU?](https://mlsanity.com/paper/2305.08414/) (2023)
- [Can Automatic Metrics Assess High-Quality Translations?](https://mlsanity.com/paper/2405.18348/) (2024)
- [DEEP: DEnoising Entity Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2111.07393/) (2021)
- [adaptMLLM: Fine-Tuning Multilingual Language Models on Low-Resource Languages with Integrated LLM Playgrounds](https://mlsanity.com/paper/2403.02370/) (2024)
- [Document-Level Language Models for Machine Translation](https://mlsanity.com/paper/2310.12303/) (2023)
- [Beyond Decoder-only: Large Language Models Can be Good Encoders for Machine Translation](https://mlsanity.com/paper/2503.06594/) (2025)
- [Discourse Centric Evaluation of Machine Translation with a Densely Annotated Parallel Corpus](https://mlsanity.com/paper/2305.11142/) (2023)
- [Large Language Models "Ad Referendum": How Good Are They at Machine Translation in the Legal Domain?](https://mlsanity.com/paper/2402.07681/) (2024)
- [Enhancing Human Evaluation in Machine Translation with Comparative Judgment](https://mlsanity.com/paper/2502.17797/) (2025)
- [An Interdisciplinary Approach to Human-Centered Machine Translation](https://mlsanity.com/paper/2506.13468/) (2025)
- [An Overview on Machine Translation Evaluation](https://mlsanity.com/paper/2202.11027/) (2022)
- [A Bilingual Parallel Corpus with Discourse Annotations](https://mlsanity.com/paper/2210.14667/) (2022)
- [Pixels and Predictions: Potential of GPT-4V in Meteorological Imagery Analysis and Forecast Communication](https://mlsanity.com/paper/2404.15166/) (2024)
- [Is It Good Data for Multilingual Instruction Tuning or Just Bad Multilingual Evaluation for Large Language Models?](https://mlsanity.com/paper/2406.12822/) (2024)
- [You Cannot Feed Two Birds with One Score: the Accuracy-Naturalness Tradeoff in Translation](https://mlsanity.com/paper/2503.24013/) (2025)
- [Are AI agents the new machine translation frontier? Challenges and opportunities of single- and multi-agent systems for multilingual digital communication](https://mlsanity.com/paper/2504.12891/) (2025)
- [On User Interfaces for Large-Scale Document-Level Human Evaluation of Machine Translation Outputs](https://mlsanity.com/paper/2104.10408/) (2021)
- [Variance-Aware Machine Translation Test Sets](https://mlsanity.com/paper/2111.04079/) (2021)
- [Searching for a higher power in the human evaluation of MT](https://mlsanity.com/paper/2210.11612/) (2022)
- [Human Evaluation of English--Irish Transformer-Based NMT](https://mlsanity.com/paper/2403.02366/) (2024)
- [Extract and Attend: Improving Entity Translation in Neural Machine Translation](https://mlsanity.com/paper/2306.02242/) (2023)
- [A Multi-way Parallel Named Entity Annotated Corpus for English, Tamil and Sinhala](https://mlsanity.com/paper/2412.02056/) (2024)
- [DOLFIN -- Document-Level Financial test set for Machine Translation](https://mlsanity.com/paper/2502.03053/) (2025)
- [Last Translation Benchmark](https://mlsanity.com/paper/2609.04173/) (2026)
- [HilMeMe: A Human-in-the-Loop Machine Translation Evaluation Metric Looking into Multi-Word Expressions](https://mlsanity.com/paper/2211.05201/) (2022)
- [Evaluation Metrics of Language Generation Models for Synthetic Traffic Generation Tasks](https://mlsanity.com/paper/2311.12534/) (2023)

## Abstract (arXiv)

The quality of machine translation has increased remarkably over the past years, to the degree that it was found to be indistinguishable from professional human translation in a number of empirical investigations. We reassess Hassan et al.'s 2018 investigation into Chinese to English news translation, showing that the finding of human-machine parity was owed to weaknesses in the evaluation design - which is currently considered best practice in the field. We show that the professional human translations contained significantly fewer errors, and that perceived quality in human evaluation depends on the choice of raters, the availability of linguistic context, and the creation of reference translations. Our results call for revisiting current best practices to assess strong machine translation systems in general and human-machine parity in particular, for which we offer a set of recommendations based on our empirical findings.