# Benchmarking GPT-4 against Human Translators: A Comprehensive Evaluation Across Languages, Domains, and Expertise Levels

arXiv: https://arxiv.org/abs/2411.13775 · HTML: https://mlsanity.com/paper/2411.13775/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 54 references that are in the map.
- First posted on arXiv 21 Nov 2024.

## Builds on (most cited first, 14 of 14)

- [Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models](https://mlsanity.com/paper/2309.01219/) (2023)
- [How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation](https://mlsanity.com/paper/2302.09210/) (2023)
- [News Summarization and Evaluation in the Era of GPT-3](https://mlsanity.com/paper/2209.12356/) (2022)
- [CLUE: A Chinese Language Understanding Evaluation Benchmark](https://mlsanity.com/paper/2004.05986/) (2020)
- [Is ChatGPT A Good Translator? Yes With GPT-4 As The Engine](https://mlsanity.com/paper/2301.08745/) (2023)
- [SeamlessM4T: Massively Multilingual & Multimodal Machine Translation](https://mlsanity.com/paper/2308.11596/) (2023)
- [Achieving Human Parity on Automatic Chinese to English News Translation](https://mlsanity.com/paper/1803.05567/) (2018)
- [Adapting Large Language Models for Document-Level Machine Translation](https://mlsanity.com/paper/2401.06468/) (2024)
- [From LLM to NMT: Advancing Low-Resource Machine Translation with Claude](https://mlsanity.com/paper/2404.13813/) (2024)
- [(Perhaps) Beyond Human Translation: Harnessing Multi-Agent Collaboration for Translating Ultra-Long Literary Texts](https://mlsanity.com/paper/2405.11804/) (2024)
- [TEaR: Improving LLM-based Machine Translation with Systematic Self-Refinement](https://mlsanity.com/paper/2402.16379/) (2024)
- [Benchmarking Large Language Models on CFLUE -- A Chinese Financial Language Understanding Evaluation Dataset](https://mlsanity.com/paper/2405.10542/) (2024)
- [Does GPT-4 surpass human performance in linguistic pragmatics?](https://mlsanity.com/paper/2312.09545/) (2023)
- [Evaluating Large Language Models for Radiology Natural Language Processing](https://mlsanity.com/paper/2307.13693/) (2023)

## Built on by (most cited first, 9 of 9)

- [MAATS: A Multi-Agent Automated Translation System Based on MQM Evaluation](https://mlsanity.com/paper/2505.14848/) (2025)
- [How Well Do Large Reasoning Models Translate? A Comprehensive Evaluation for Multi-Domain Machine Translation](https://mlsanity.com/paper/2505.19987/) (2025)
- [Parallel Corpora for Machine Translation in Low-resource Indic Languages: A Comprehensive Review](https://mlsanity.com/paper/2503.04797/) (2025)
- [DITING: A Multi-Agent Evaluation Framework for Benchmarking Web Novel Translation](https://mlsanity.com/paper/2510.09116/) (2025)
- [Are Large Reasoning Models Good Translation Evaluators? Analysis and Performance Boost](https://mlsanity.com/paper/2510.20780/) (2025)
- [Dual Debiasing for Noisy In-Context Learning for Text Generation](https://mlsanity.com/paper/2506.00418/) (2025)
- [Measuring Intent Comprehension in LLMs](https://mlsanity.com/paper/2506.16584/) (2025)
- [Automated evaluation of LLMs for effective machine translation of Mandarin Chinese to English](https://mlsanity.com/paper/2603.09998/) (2026)
- [To Write or to Automate Linguistic Prompts, That Is the Question](https://mlsanity.com/paper/2603.25169/) (2026)

## Abstract (arXiv)

This study presents a comprehensive evaluation of GPT-4's translation capabilities compared to human translators of varying expertise levels. Through systematic human evaluation using the MQM schema, we assess translations across three language pairs (ChineselongleftrightarrowEnglish, RussianlongleftrightarrowEnglish, and ChineselongleftrightarrowHindi) and three domains (News, Technology, and Biomedical). Our findings reveal that GPT-4 achieves performance comparable to junior-level translators in terms of total errors, while still lagging behind senior translators. Unlike traditional Neural Machine Translation systems, which show significant performance degradation in resource-poor language directions, GPT-4 maintains consistent translation quality across all evaluated language pairs. Through qualitative analysis, we identify distinctive patterns in translation approaches: GPT-4 tends toward overly literal translations and exhibits lexical inconsistency, while human translators sometimes over-interpret context and introduce hallucinations. This study represents the first systematic comparison between LLM and human translators across different proficiency levels, providing valuable insights into the current capabilities and limitations of LLM-based translation systems.