mlsanityData to 14 Oct 2026

GPT-4 vs. Human Translators: A Comprehensive Evaluation of Translation Quality Across Languages, Domains, and Expertise Levels

arXiv 2407.03658 ↗First posted 4 Jul 2024cs.CLcited by 21

Not on the map. Why?

Abstract (arXiv)

This study comprehensively evaluates the translation quality of Large Language Models (LLMs), specifically GPT-4, against human translators of varying expertise levels across multiple language pairs and domains. Through carefully designed annotation rounds, we find that GPT-4 performs comparably to junior translators in terms of total errors made but lags behind medium and senior translators. We also observe the imbalanced performance across different languages and domains, with GPT-4's translation capability gradually weakening from resource-rich to resource-poor directions. In addition, we qualitatively study the translation given by GPT-4 and human translators, and find that GPT-4 translator suffers from literal translations, but human translators sometimes overthink the background information. To our knowledge, this study is the first to evaluate LLMs against human translators and analyze the systematic differences between their outputs, providing valuable insights into the current state of LLM-based translation and its potential limitations.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 18 of its 50 references that are in the map. First posted on arXiv 4 Jul 2024.

Builds on

18 of 50 references.

References · 18

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  2. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  3. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  4. 2023How Good Are GPT Models at Machine Translation? A Comprehensive Evaluationcited by 348
  5. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 338
  6. 2022News Summarization and Evaluation in the Era of GPT-3cited by 327
  7. 2023Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvaturecited by 287
  8. 2020CLUE: A Chinese Language Understanding Evaluation Benchmarkcited by 278
  9. 2023Is ChatGPT A Good Translator? Yes With GPT-4 As The Enginecited by 176
  10. 2023Towards Making the Most of ChatGPT for Machine Translationcited by 138
  11. 2024Benchmarking LLMs via Uncertainty Quantificationcited by 126
  12. 2018Achieving Human Parity on Automatic Chinese to English News Translationcited by 119
6 more
  1. 2024Adapting Large Language Models for Document-Level Machine Translationcited by 59
  2. 2024(Perhaps) Beyond Human Translation: Harnessing Multi-Agent Collaboration for Translating Ultra-Long Literary Textscited by 49
  3. 2024TEaR: Improving LLM-based Machine Translation with Systematic Self-Refinementcited by 39
  4. 2023Does GPT-4 surpass human performance in linguistic pragmatics?cited by 16
  5. 2018Quantitative Fine-Grained Human Evaluation of Machine Translation Systems: a Case Study on English to Croatiancited by 7
  6. 2020What's the Difference Between Professional Human and Machine Translation? A Blind Multi-language Study on Domain-specific MTcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 12; 2026: 3.

21 papers.

Sort:

By area:

  1. 2025Multilingual != Multicultural: Evaluating Gaps Between Multilingual Capabilities and Cultural Alignment in LLMscited by 39
  2. 2024How Good Are LLMs for Literary Translation, Really? Literary Translation Evaluation with Humans and LLMscited by 23
  3. 2024See What LLMs Cannot Answer: A Self-Challenge Framework for Uncovering LLM Weaknessescited by 15
  4. 2025Cross-lingual Collapse: How Language-Centric Foundation Models Shape Reasoning in Large Language Modelscited by 11
  5. 2024System 2 thinking in OpenAI's o1-preview model: Near-perfect performance on a mathematics examcited by 10
  6. 2025LiTransProQA: an LLM-based Literary Translation evaluation metric with Professional Question Answeringcited by 10
  7. 2025MAPS: A Multilingual Benchmark for Agent Performance and Securitycited by 7
  8. 2025WETBench: A Benchmark for Detecting Task-Specific Machine-Generated Text on Wikipediacited by 6
  9. 2025How Well Do Large Reasoning Models Translate? A Comprehensive Evaluation for Multi-Domain Machine Translationcited by 5
  10. 2025Will It Still Be True Tomorrow? Multilingual Evergreen Question Classification to Improve Trustworthy QAcited by 5
  11. 2024Beyond Human-Only: Evaluating Human-Machine Collaboration for Collecting High-Quality Translation Datacited by 4
  12. 2025Multilingual JobBERT for Cross-Lingual Job Title Matchingcited by 4
9 more
  1. 2025MULTITAT: Benchmarking Multilingual Table-and-Text Question Answeringcited by 2
  2. 2025Bangla-Bayanno: A 52K-Pair Bengali Visual Question Answering Dataset with LLM-Assisted Translation Refinementcited by 2
  3. 2024The use of GPT-4o and Other Large Language Models for the Improvement and Design of Self-Assessment Scales for Measurement of Interpersonal Communication Skillscited by 1
  4. 2025Measuring Intent Comprehension in LLMscited by 1
  5. 2025Ready to Translate, Not to Represent? Bias and Performance Gaps in Multilingual LLMs Across Language Families and Domainscited by 1
  6. 2026Cross-Lingual Attention Distillation with Personality-Informed Generative Augmentation for Multilingual Personality Recognitioncited by 1
  7. 2024SandboxAQ's submission to MRL 2024 Shared Task on Multi-lingual Multi-task Information Retrievalcited by 0
  8. 2026Automated evaluation of LLMs for effective machine translation of Mandarin Chinese to Englishcited by 0
  9. 2026The Language-Energy Divide: Measuring Energy Costs of Multilingual LLM Inferencecited by 0