mlsanityData to 7 Oct 2026

ChatGPT MT: Competitive for High- (but not Low-) Resource Languages

arXiv 2309.07423 ↗First posted 14 Sep 2023cs.CLcited by 23

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) implicitly learn to perform a range of language tasks, including machine translation (MT). Previous studies explore aspects of LLMs' MT capabilities. However, there exist a wide variety of languages for which recent LLM MT performance has never before been evaluated. Without published experimental evidence on the matter, it is difficult for speakers of the world's diverse languages to know how and whether they can use LLMs for their languages. We present the first experimental evidence for an expansive set of 204 languages, along with MT cost analysis, using the FLORES-200 benchmark. Trends reveal that GPT models approach or exceed traditional MT model performance for some high-resource languages (HRLs) but consistently lag for low-resource languages (LRLs), under-performing traditional MT for 84.1% of languages we covered. Our analysis reveals that a language's resource level is the most important feature in determining ChatGPT's relative ability to translate it, and suggests that ChatGPT is especially disadvantaged for LRLs and African languages.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 19 of its 24 references that are in the map. First posted on arXiv 14 Sep 2023.

Builds on

19 of 24 references.

References · 19

  1. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  2. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  3. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  4. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  5. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  6. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  7. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
  8. 2023How Good Are GPT Models at Machine Translation? A Comprehensive Evaluationcited by 344
  9. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 186
  10. 2023Is ChatGPT A Good Translator? Yes With GPT-4 As The Enginecited by 172
  11. 2023Multilingual Machine Translation with Large Language Models: Empirical Results and Analysiscited by 156
  12. 2022Towards a Cleaner Document-Oriented Multilingual Crawled Corpuscited by 143
7 more
  1. 2023M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Modelscited by 109
  2. 2023Document-Level Machine Translation with Large Language Modelscited by 90
  3. 2022Building Machine Translation Systems for the Next Thousand Languagescited by 69
  4. 2023Adaptive Machine Translation with Large Language Modelscited by 58
  5. 2023SuperCLUE: A Comprehensive Chinese Large Language Model Benchmarkcited by 50
  6. 2023How to Design Translation Prompts for ChatGPT: An Empirical Studycited by 31
  7. 2022Towards the Next 1000 Languages in Multilingual Machine Translation: Exploring the Synergy Between Supervised and Self-Supervised Learningcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 10; 2025: 8; 2026: 4.

23 papers.

Sort:

By area:

  1. 2024Many-Shot In-Context Learningcited by 203
  2. 2024What Are Tools Anyway? A Survey from the Language Model Perspectivecited by 53
  3. 2024(Perhaps) Beyond Human Translation: Harnessing Multi-Agent Collaboration for Translating Ultra-Long Literary Textscited by 49
  4. 2025Crosslingual Reasoning through Test-Time Scalingcited by 46
  5. 2023An In-depth Look at Gemini's Language Abilitiescited by 28
  6. 2024How Much Data is Enough Data? Fine-Tuning Large Language Models for In-House Translation: Performance Evaluation Across Multiple Dataset Sizescited by 17
  7. 2025New Trends for Modern Machine Translation with Large Reasoning Modelscited by 17
  8. 2024On Subjective Uncertainty Quantification and Calibration in Natural Language Generationcited by 14
  9. 2025Bridging the Linguistic Divide: A Survey on Leveraging Large Language Models for Machine Translationcited by 8
  10. 2025mSTEB: Massively Multilingual Evaluation of LLMs on Speech and Text Taskscited by 7
  11. 2025The World According to LLMs: How Geographic Origin Influences LLMs' Entity Deduction Capabilitiescited by 5
  12. 2024Mufu: Multilingual Fused Learning for Low-Resource Translation with LLMcited by 3
11 more
  1. 2024Domain-Specific Translation with Open-Source Large Language Models: Resource-Oriented Analysiscited by 3
  2. 2025OWL: Probing Cross-Lingual Recall of Memorized Texts via World Literaturecited by 3
  3. 2025TASER: Translation Assessment via Systematic Evaluation and Reasoningcited by 3
  4. 2024Towards Chapter-to-Chapter Context-Aware Literary Translation via Large Language Modelscited by 2
  5. 2024Evaluating the Translation Performance of Large Language Models Based on Euas-20cited by 2
  6. 2024A Comparative Study of Translation Bias and Accuracy in Multilingual Large Language Models for Cross-Language Claim Verificationcited by 1
  7. 2025Sparse Subnetwork Enhancement for Underrepresented Languages in Large Language Modelscited by 1
  8. 2026Reflective Translation: Improving Low-Resource Machine Translation via Structured Self-Reflectioncited by 0
  9. 2026QFS-Composer: Query-focused summarization pipeline for less resourced languagescited by 0
  10. 2026Nsanku: Evaluating Zero-Shot Translation Performance of LLMs for Ghanaian Languagescited by 0
  11. 2026Evaluating Communicative Success in Machine-Translated Conversationcited by 0