# ChatGPT MT: Competitive for High- (but not Low-) Resource Languages

arXiv: https://arxiv.org/abs/2309.07423 · HTML: https://mlsanity.com/paper/2309.07423/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 24 references that are in the map.
- First posted on arXiv 14 Sep 2023.

## Builds on (most cited first, 19 of 19)

- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [Beyond English-Centric Multilingual Machine Translation](https://mlsanity.com/paper/2010.11125/) (2020)
- [The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation](https://mlsanity.com/paper/2106.03193/) (2021)
- [How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation](https://mlsanity.com/paper/2302.09210/) (2023)
- [Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models](https://mlsanity.com/paper/2305.13707/) (2023)
- [Is ChatGPT A Good Translator? Yes With GPT-4 As The Engine](https://mlsanity.com/paper/2301.08745/) (2023)
- [Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis](https://mlsanity.com/paper/2304.04675/) (2023)
- [Towards a Cleaner Document-Oriented Multilingual Crawled Corpus](https://mlsanity.com/paper/2201.06642/) (2022)
- [M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Models](https://mlsanity.com/paper/2306.05179/) (2023)
- [Document-Level Machine Translation with Large Language Models](https://mlsanity.com/paper/2304.02210/) (2023)
- [Building Machine Translation Systems for the Next Thousand Languages](https://mlsanity.com/paper/2205.03983/) (2022)
- [Adaptive Machine Translation with Large Language Models](https://mlsanity.com/paper/2301.13294/) (2023)
- [SuperCLUE: A Comprehensive Chinese Large Language Model Benchmark](https://mlsanity.com/paper/2307.15020/) (2023)
- [How to Design Translation Prompts for ChatGPT: An Empirical Study](https://mlsanity.com/paper/2304.02182/) (2023)
- [Towards the Next 1000 Languages in Multilingual Machine Translation: Exploring the Synergy Between Supervised and Self-Supervised Learning](https://mlsanity.com/paper/2201.03110/) (2022)

## Built on by (most cited first, 23 of 23)

- [Many-Shot In-Context Learning](https://mlsanity.com/paper/2404.11018/) (2024)
- [What Are Tools Anyway? A Survey from the Language Model Perspective](https://mlsanity.com/paper/2403.15452/) (2024)
- [(Perhaps) Beyond Human Translation: Harnessing Multi-Agent Collaboration for Translating Ultra-Long Literary Texts](https://mlsanity.com/paper/2405.11804/) (2024)
- [Crosslingual Reasoning through Test-Time Scaling](https://mlsanity.com/paper/2505.05408/) (2025)
- [An In-depth Look at Gemini's Language Abilities](https://mlsanity.com/paper/2312.11444/) (2023)
- [How Much Data is Enough Data? Fine-Tuning Large Language Models for In-House Translation: Performance Evaluation Across Multiple Dataset Sizes](https://mlsanity.com/paper/2409.03454/) (2024)
- [New Trends for Modern Machine Translation with Large Reasoning Models](https://mlsanity.com/paper/2503.10351/) (2025)
- [On Subjective Uncertainty Quantification and Calibration in Natural Language Generation](https://mlsanity.com/paper/2406.05213/) (2024)
- [Bridging the Linguistic Divide: A Survey on Leveraging Large Language Models for Machine Translation](https://mlsanity.com/paper/2504.01919/) (2025)
- [mSTEB: Massively Multilingual Evaluation of LLMs on Speech and Text Tasks](https://mlsanity.com/paper/2506.08400/) (2025)
- [The World According to LLMs: How Geographic Origin Influences LLMs' Entity Deduction Capabilities](https://mlsanity.com/paper/2508.05525/) (2025)
- [OWL: Probing Cross-Lingual Recall of Memorized Texts via World Literature](https://mlsanity.com/paper/2505.22945/) (2025)
- [Mufu: Multilingual Fused Learning for Low-Resource Translation with LLM](https://mlsanity.com/paper/2409.13949/) (2024)
- [Domain-Specific Translation with Open-Source Large Language Models: Resource-Oriented Analysis](https://mlsanity.com/paper/2412.05862/) (2024)
- [TASER: Translation Assessment via Systematic Evaluation and Reasoning](https://mlsanity.com/paper/2510.00255/) (2025)
- [Towards Chapter-to-Chapter Context-Aware Literary Translation via Large Language Models](https://mlsanity.com/paper/2407.08978/) (2024)
- [Evaluating the Translation Performance of Large Language Models Based on Euas-20](https://mlsanity.com/paper/2408.03119/) (2024)
- [A Comparative Study of Translation Bias and Accuracy in Multilingual Large Language Models for Cross-Language Claim Verification](https://mlsanity.com/paper/2410.10303/) (2024)
- [Sparse Subnetwork Enhancement for Underrepresented Languages in Large Language Models](https://mlsanity.com/paper/2510.13580/) (2025)
- [Reflective Translation: Improving Low-Resource Machine Translation via Structured Self-Reflection](https://mlsanity.com/paper/2601.19871/) (2026)
- [QFS-Composer: Query-focused summarization pipeline for less resourced languages](https://mlsanity.com/paper/2604.10687/) (2026)
- [Nsanku: Evaluating Zero-Shot Translation Performance of LLMs for Ghanaian Languages](https://mlsanity.com/paper/2605.04208/) (2026)
- [Evaluating Communicative Success in Machine-Translated Conversation](https://mlsanity.com/paper/2609.19885/) (2026)

## Abstract (arXiv)

Large language models (LLMs) implicitly learn to perform a range of language tasks, including machine translation (MT). Previous studies explore aspects of LLMs' MT capabilities. However, there exist a wide variety of languages for which recent LLM MT performance has never before been evaluated. Without published experimental evidence on the matter, it is difficult for speakers of the world's diverse languages to know how and whether they can use LLMs for their languages. We present the first experimental evidence for an expansive set of 204 languages, along with MT cost analysis, using the FLORES-200 benchmark. Trends reveal that GPT models approach or exceed traditional MT model performance for some high-resource languages (HRLs) but consistently lag for low-resource languages (LRLs), under-performing traditional MT for 84.1% of languages we covered. Our analysis reveals that a language's resource level is the most important feature in determining ChatGPT's relative ability to translate it, and suggests that ChatGPT is especially disadvantaged for LRLs and African languages.