mlsanityData to 7 Oct 2026

How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation

arXiv 2302.09210 ↗First posted 18 Feb 2023cs.CLcited by 344

Not on the map. Why?

Cited by 25 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Generative Pre-trained Transformer (GPT) models have shown remarkable capabilities for natural language generation, but their performance for machine translation has not been thoroughly investigated. In this paper, we present a comprehensive evaluation of GPT models for machine translation, covering various aspects such as quality of different GPT models in comparison with state-of-the-art research and commercial systems, effect of prompting strategies, robustness towards domain shifts and document-level translation. We experiment with eighteen different translation directions involving high and low resource languages, as well as non English-centric translations, and evaluate the performance of three GPT models: ChatGPT, GPT3.5 (text-davinci-003), and text-davinci-002. Our results show that GPT models achieve very competitive translation quality for high resource languages, while having limited capabilities for low resource languages. We also show that hybrid approaches, which combine GPT models with other translation systems, can further enhance the translation quality. We perform comprehensive analysis and human evaluation to further understand the characteristics of GPT translations. We hope that our paper provides valuable insights for researchers and practitioners in the field and helps to better understand the potential and limitations of GPT models for translation.

Key facts. Cited by 344 papers in the map (data through 7 Oct 2026). Builds on 25 of its 36 references that are in the map. First posted on arXiv 18 Feb 2023.

Builds on

25 of 36 references.

References · 25

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  8. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  9. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  10. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  11. 2020Language-agnostic BERT Sentence Embeddingcited by 621
  12. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
13 more
  1. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  2. 2022News Summarization and Evaluation in the Era of GPT-3cited by 324
  3. 2023Prompting Large Language Model for Machine Translation: A Case Studycited by 247
  4. 2021Word Alignment by Fine-tuning Embeddings on Parallel Corporacited by 186
  5. 2022In-context Examples Selection for Machine Translationcited by 172
  6. 2021The Curious Case of Hallucinations in Neural Machine Translationcited by 161
  7. 2022Prompting PaLM for Translation: Assessing Strategies and Performancecited by 144
  8. 2023The unreasonable effectiveness of few-shot learning for machine translationcited by 91
  9. 2021Scalable and Efficient MoE Training for Multitask Multilingual Modelscited by 76
  10. 2019The Effect of Translationese in Machine Translation Test Setscited by 49
  11. 2020Rethinking Document-level Neural Machine Translationcited by 32
  12. 2022SALTED: A Framework for SAlient Long-Tail Translation Error Detectioncited by 23
  13. 2022Finding Memo: Extractive Memorization in Constrained Sequence Generation Taskscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 102; 2024: 136; 2025: 72; 2026: 34.

344 papers.

Sort:

By area:

  1. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 995
  2. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 416
  3. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translationcited by 390
  4. 2023MEGA: Multilingual Evaluation of Generative AIcited by 291
  5. 2023Low-Resource Languages Jailbreak GPT-4cited by 290
  6. 2023A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Modelscited by 244
  7. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 224
  8. 2024Tower: An Open Multilingual Large Language Model for Translation-Related Taskscited by 205
  9. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 203
  10. 2023Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Promptingcited by 163
  11. 2023Multilingual Machine Translation with Large Language Models: Empirical Results and Analysiscited by 156
  12. 2023Towards Making the Most of ChatGPT for Machine Translationcited by 138
38 more
  1. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 138
  2. 2023Hallucinations in Large Multilingual Translation Modelscited by 126
  3. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
  4. 2023DatasetDM: Synthesizing Data with Perception Annotations Using Diffusion Modelscited by 122
  5. 2024Language Model Cascades: Token-level uncertainty and beyondcited by 115
  6. 2023A Survey of Text Watermarking in the Era of Large Language Modelscited by 102
  7. 2023Graph Foundation Models: Concepts, Opportunities and Challengescited by 101
  8. 2023Is ChatGPT a Highly Fluent Grammatical Error Correction System? A Comprehensive Evaluationcited by 97
  9. 2023Cross-lingual Prompting: Improving Zero-shot Chain-of-Thought Reasoning across Languagescited by 96
  10. 2023Document-Level Machine Translation with Large Language Modelscited by 90
  11. 2023A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Modelscited by 88
  12. 2023Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Modelscited by 87
  13. 2023Do Multilingual Language Models Think Better in English?cited by 87
  14. 2023Cognitive Mirage: A Review of Hallucinations in Large Language Modelscited by 83
  15. 2023Large language models effectively leverage document-level context for literary translation, but critical errors persistcited by 80
  16. 2025Universal Model Routing for Efficient LLM Inferencecited by 80
  17. 2023GPTEval: A Survey on Assessments of ChatGPT and GPT-4cited by 73
  18. 2023BigTranslate: Augmenting Large Language Models with Multilingual Translation Capability over 100 Languagescited by 60
  19. 2024Adapting Large Language Models for Document-Level Machine Translationcited by 59
  20. 2023How Robust is GPT-3.5 to Predecessors? A Comprehensive Study on Language Understanding Taskscited by 58
  21. 2023Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOMcited by 58
  22. 2025RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMscited by 58
  23. 2023MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities, Models and Taskscited by 57
  24. 2024Revisiting Demonstration Selection Strategies in In-Context Learningcited by 57
  25. 2023Leveraging GPT-4 for Automatic Translation Post-Editingcited by 56
  26. 2023Generative Pre-trained Transformer: A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directionscited by 55
  27. 2023Eliciting the Translation Ability of Large Language Models via Multilingual Finetuning with Translation Instructionscited by 55
  28. 2023Benchmarking Machine Translation with Cultural Awarenesscited by 53
  29. 2023TIM: Teaching Large Language Models to Translate with Comparisoncited by 50
  30. 2023Steering Large Language Models for Machine Translation with Finetuning and In-Context Learningcited by 48
  31. 2024LLaMAX: Scaling Linguistic Horizons of LLM by Enhancing Translation Capabilities Beyond 100 Languagescited by 47
  32. 2023ChatGPT: Beginning of an End of Manual Linguistic Data Annotation? Use Case of Automatic Genre Identificationcited by 45
  33. 2023Democratizing LLMs for Low-Resource Languages by Leveraging their English Dominant Abilities with Linguistically-Diverse Promptscited by 45
  34. 2024Ethical Reasoning and Moral Value Alignment of LLMs Depend on the Language we Prompt them incited by 45
  35. 2023Exploring Human-Like Translation Strategy with Large Language Modelscited by 44
  36. 2024Fine-tuning Large Language Models for Domain-specific Machine Translationcited by 43
  37. 2024IrokoBench: A New Benchmark for African Languages in the Age of Large Language Modelscited by 43
  38. 2023Mitigating Hallucinations and Off-target Machine Translation with Source-Contrastive and Language-Contrastive Decodingcited by 42

50 most cited of 344 All 344 (JSON)