mlsanityData to 14 Oct 2026

Attaining the Unattainable? Reassessing Claims of Human Parity in Neural Machine Translation

arXiv 1808.10432 ↗First posted 30 Aug 2018cs.CLcited by 58

Not on the map. Why?

Abstract (arXiv)

We reassess a recent study (Hassan et al., 2018) that claimed that machine translation (MT) has reached human parity for the translation of news from Chinese into English, using pairwise ranking and considering three variables that were not taken into account in that previous study: the language in which the source side of the test set was originally written, the translation proficiency of the evaluators, and the provision of inter-sentential context. If we consider only original source text (i.e. not translated from another language, or translationese), then we find evidence showing that human parity has not been achieved. We compare the judgments of professional translators against those of non-experts and discover that those of the experts result in higher inter-annotator agreement and better discrimination between human and machine translations. In addition, we analyse the human translations of the test set and identify important translation issues. Finally, based on these findings, we provide a set of recommendations for future human evaluations of MT.

Key facts. Cited by 58 papers in the map (data through 14 Oct 2026). First posted on arXiv 30 Aug 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 23; 2022: 10; 2023: 8; 2024: 10; 2025: 6; 2026: 1.

58 papers.

Sort:

  1. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 338
  2. 2022Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Datasetcited by 100
  3. 2022A Few Thousand Translations Go a Long Way! Leveraging Pre-trained Models for African News Translationcited by 98
  4. 2021The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generationcited by 97
  5. 2021Survey of Low-Resource Machine Translationcited by 92
  6. 2022The Fallacy of AI Functionalitycited by 88
  7. 2021Machine Translationese: Effects of Algorithmic Bias on Linguistic Complexity in Machine Translationcited by 70
  8. 2023Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOMcited by 59
  9. 2022Democratizing Neural Machine Translation with OPUS-MTcited by 58
  10. 2021Measuring and Increasing Context Usage in Context-Aware Machine Translationcited by 48
  11. 2024Navigating the Metrics Maze: Reconciling Score Magnitudes and Accuraciescited by 40
  12. 2021The Dangers of Underclaiming: Reasons for Caution When Reporting How NLP Systems Failcited by 38
38 more
  1. 2021When Does Translation Require Context? A Data-driven, Multilingual Explorationcited by 36
  2. 2024A Shocking Amount of the Web is Machine Translated: Insights from Multi-Way Parallelismcited by 35
  3. 2021Bidimensional Leaderboards: Generate and Evaluate Language Hand in Handcited by 32
  4. 2023Iterative Translation Refinement with Large Language Modelscited by 27
  5. 2024How Good Are LLMs for Literary Translation, Really? Literary Translation Evaluation with Humans and LLMscited by 23
  6. 2021Do Context-Aware Translation Models Pay the Right Attention?cited by 21
  7. 2021GENIE: Toward Reproducible and Standardized Human Evaluation for Text Generationcited by 20
  8. 2021The statistical advantage of automatic NLG metrics at the system levelcited by 18
  9. 2021Modeling Bilingual Conversational Characteristics for Neural Chat Translationcited by 18
  10. 2021Human Evaluation of Creative NLG Systems: An Interdisciplinary Survey on Recent Paperscited by 18
  11. 2021Bandits Don't Follow Rules: Balancing Multi-Facet Machine Translation with Multi-Armed Banditscited by 16
  12. 2024Can Automatic Metrics Assess High-Quality Translations?cited by 16
  13. 2023Identifying Context-Dependent Translations for Evaluation Set Productioncited by 14
  14. 2021Comparing Feature-Engineering and Feature-Learning Approaches for Multilingual Translationese Classificationcited by 13
  15. 2021Towards Making the Most of Dialogue Characteristics for Neural Chat Translationcited by 12
  16. 2022Original or Translated? A Causal Analysis of the Impact of Translationese on Machine Translation Performancecited by 11
  17. 2021A Comparison of Approaches to Document-level Machine Translationcited by 7
  18. 2023Neural Machine Translation for the Indigenous Languages of the Americas: An Introductioncited by 7
  19. 2023Quantifying the Plausibility of Context Reliance in Neural Machine Translationcited by 7
  20. 2025Enhancing Human Evaluation in Machine Translation with Comparative Judgmentcited by 7
  21. 2024Pretraining Language Models Using Translationesecited by 6
  22. 2022Explaining Translationese: why are Neural Classifiers Better and what do they Learn?cited by 5
  23. 2023Translating away Translationese without Parallel Datacited by 5
  24. 2023On-the-Fly Fusion of Large Language Models and Machine Translationcited by 5
  25. 2024Towards Tailored Recovery of Lexical Diversity in Literary Machine Translationcited by 5
  26. 2021Automatic Classification of Human Translation and Machine Translation: A Study from the Perspective of Lexical Diversitycited by 4
  27. 2021On Systematic Style Differences between Unsupervised and Supervised MT and an Application for High-Resource Machine Translationcited by 4
  28. 2021Fair Comparison: Quantifying Variance in Resultsfor Fine-grained Visual Categorizationcited by 4
  29. 2024M3T: A New Benchmark Dataset for Multi-Modal Document-Level Machine Translationcited by 4
  30. 2021On User Interfaces for Large-Scale Document-Level Human Evaluation of Machine Translation Outputscited by 3
  31. 2021Variance-Aware Machine Translation Test Setscited by 3
  32. 2022Towards Debiasing Translation Artifactscited by 3
  33. 2022Searching for a higher power in the human evaluation of MTcited by 3
  34. 2023Chat Translation Error Detection for Assisting Cross-lingual Communicationscited by 3
  35. 2024Human Evaluation of English--Irish Transformer-Based NMTcited by 3
  36. 2024MQM-Chat: Multidimensional Quality Metrics for Chat Translationcited by 3
  37. 2025Long-context Reference-based MT Quality Estimationcited by 2
  38. 2022Discourse Cohesion Evaluation for Document-Level Neural Machine Translationcited by 1

50 most cited of 58 All 58 (JSON)