mlsanityData to 7 Oct 2026

A Set of Recommendations for Assessing Human-Machine Parity in Language Translation

arXiv 2004.01694 ↗First posted 3 Apr 2020cs.CL cs.AIcited by 34

Not on the map. Why?

Abstract (arXiv)

The quality of machine translation has increased remarkably over the past years, to the degree that it was found to be indistinguishable from professional human translation in a number of empirical investigations. We reassess Hassan et al.'s 2018 investigation into Chinese to English news translation, showing that the finding of human-machine parity was owed to weaknesses in the evaluation design - which is currently considered best practice in the field. We show that the professional human translations contained significantly fewer errors, and that perceived quality in human evaluation depends on the choice of raters, the availability of linguistic context, and the creation of reference translations. Our results call for revisiting current best practices to assess strong machine translation systems in general and human-machine parity in particular, for which we offer a set of recommendations based on our empirical findings.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 4; 2023: 5; 2024: 7; 2025: 8; 2026: 3.

34 papers.

Sort:

By area:

  1. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 333
  2. 2021Scientific Credibility of Machine Translation Research: A Meta-Evaluation of 769 Paperscited by 70
  3. 2021BlonDe: An Automatic Evaluation Metric for Document-level Machine Translationcited by 58
  4. 2021Translation Quality Assessment: A Brief Survey on Manual and Automatic Methodscited by 23
  5. 2023What's the Meaning of Superhuman Performance in Today's NLU?cited by 22
  6. 2024Can Automatic Metrics Assess High-Quality Translations?cited by 16
  7. 2021DEEP: DEnoising Entity Pre-training for Neural Machine Translationcited by 15
  8. 2024adaptMLLM: Fine-Tuning Multilingual Language Models on Low-Resource Languages with Integrated LLM Playgroundscited by 14
  9. 2023Document-Level Language Models for Machine Translationcited by 12
  10. 2025Beyond Decoder-only: Large Language Models Can be Good Encoders for Machine Translationcited by 8
  11. 2023Discourse Centric Evaluation of Machine Translation with a Densely Annotated Parallel Corpuscited by 7
  12. 2024Large Language Models "Ad Referendum": How Good Are They at Machine Translation in the Legal Domain?cited by 7
22 more
  1. 2025Enhancing Human Evaluation in Machine Translation with Comparative Judgmentcited by 7
  2. 2025An Interdisciplinary Approach to Human-Centered Machine Translationcited by 6
  3. 2022An Overview on Machine Translation Evaluationcited by 4
  4. 2024Pixels and Predictions: Potential of GPT-4V in Meteorological Imagery Analysis and Forecast Communicationcited by 4
  5. 2024Is It Good Data for Multilingual Instruction Tuning or Just Bad Multilingual Evaluation for Large Language Models?cited by 4
  6. 2025You Cannot Feed Two Birds with One Score: the Accuracy-Naturalness Tradeoff in Translationcited by 4
  7. 2025Are AI agents the new machine translation frontier? Challenges and opportunities of single- and multi-agent systems for multilingual digital communicationcited by 4
  8. 2021On User Interfaces for Large-Scale Document-Level Human Evaluation of Machine Translation Outputscited by 3
  9. 2021Variance-Aware Machine Translation Test Setscited by 3
  10. 2022Searching for a higher power in the human evaluation of MTcited by 3
  11. 2022A Bilingual Parallel Corpus with Discourse Annotationscited by 3
  12. 2024Human Evaluation of English--Irish Transformer-Based NMTcited by 3
  13. 2023Extract and Attend: Improving Entity Translation in Neural Machine Translationcited by 2
  14. 2024A Multi-way Parallel Named Entity Annotated Corpus for English, Tamil and Sinhalacited by 1
  15. 2025DOLFIN -- Document-Level Financial test set for Machine Translationcited by 1
  16. 2026Last Translation Benchmarkcited by 1
  17. 2022HilMeMe: A Human-in-the-Loop Machine Translation Evaluation Metric Looking into Multi-Word Expressionscited by 0
  18. 2023Evaluation Metrics of Language Generation Models for Synthetic Traffic Generation Taskscited by 0
  19. 2025AI agents may be worth the hype but not the resources (yet): An initial exploration of machine translation quality and costs in three language pairs in the legal and news domainscited by 0
  20. 2025An Empirical Study on Chinese Character Decomposition in Multiword Expression-Aware Neural Machine Translationcited by 0
  21. 2026AI-assisted cultural heritage dissemination: Comparing NMT and glossary-augmented LLM translation in rock art documentscited by 0
  22. 2026The Almost Intelligent Revolution: Options for Scaling Up Deliberation and Empowering People with AIcited by 0