mlsanityData to 7 Oct 2026

ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning

arXiv 2304.05613 ↗First posted 12 Apr 2023cs.CL cs.AIcited by 203

Not on the map. Why?

Abstract (arXiv)

Over the last few years, large language models (LLMs) have emerged as the most important breakthroughs in natural language processing (NLP) that fundamentally transform research and developments in the field. ChatGPT represents one of the most exciting LLM systems developed recently to showcase impressive skills for language generation and highly attract public attention. Among various exciting applications discovered for ChatGPT in English, the model can process and generate texts for multiple languages due to its multilingual training data. Given the broad adoption of ChatGPT for English in different problems and areas, a natural question is whether ChatGPT can also be applied effectively for other languages or it is necessary to develop more language-specific technologies. The answer to this question requires a thorough evaluation of ChatGPT over multiple tasks with diverse languages and large datasets (i.e., beyond reported anecdotes), which is still missing or limited in current research. Our work aims to fill this gap for the evaluation of ChatGPT and similar LLMs to provide more comprehensive information for multilingual NLP applications. While this work will be an ongoing effort to include additional experiments in the future, our current paper evaluates ChatGPT on 7 different tasks, covering 37 diverse languages with high, medium, low, and extremely low resources. We also focus on the zero-shot learning setting for ChatGPT to improve reproducibility and better simulate the interactions of general users. Compared to the performance of previous models, our extensive experimental results demonstrate a worse performance of ChatGPT for different NLP tasks and languages, calling for further research to develop better models and understanding for multilingual learning.

Key facts. Cited by 203 papers in the map (data through 7 Oct 2026). Builds on 78 of its 116 references that are in the map. First posted on arXiv 12 Apr 2023.

Builds on

78 of 116 references.

References · 78

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  8. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  9. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  10. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  11. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  12. 2017Deep reinforcement learning from human preferencescited by 3,846
38 more
  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  2. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  3. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  4. 2022Emergent Abilities of Large Language Modelscited by 2,483
  5. 2018Deep contextualized word representationscited by 2,285
  6. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  7. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  8. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  9. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  10. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  11. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  12. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  13. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  14. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  15. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  16. 2019How multilingual is Multilingual BERT?cited by 642
  17. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  18. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 486
  19. 2017Word Translation Without Parallel Datacited by 480
  20. 2023How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detectioncited by 472
  21. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
  22. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 416
  23. 2019CamemBERT: a Tasty French Language Modelcited by 368
  24. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 346
  25. 2023How Good Are GPT Models at Machine Translation? A Comprehensive Evaluationcited by 344
  26. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
  27. 2017Unsupervised Machine Translation Using Monolingual Corpora Onlycited by 272
  28. 2023ChatIE: Zero-Shot Information Extraction via Chatting with ChatGPTcited by 250
  29. 2020XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generationcited by 221
  30. 2023On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspectivecited by 208
  31. 2020Are All Languages Created Equal in Multilingual BERT?cited by 206
  32. 2013Exploiting Similarities among Languages for Machine Translationcited by 187
  33. 2019Cross-Lingual Ability of Multilingual BERT: An Empirical Studycited by 182
  34. 2023Is ChatGPT A Good Translator? Yes With GPT-4 As The Enginecited by 172
  35. 2023Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERTcited by 155
  36. 2017A Survey Of Cross-lingual Word Embedding Modelscited by 142
  37. 2023Exploring the Limits of ChatGPT for Query or Aspect-based Text Summarizationcited by 122
  38. 2021Mr. TyDi: A Multi-lingual Benchmark for Dense Retrievalcited by 118

50 most cited of 78 All 78 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 55; 2024: 82; 2025: 47; 2026: 19.

203 papers.

Sort:

By area:

  1. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  2. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  3. 2023Low-Resource Languages Jailbreak GPT-4cited by 290
  4. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 224
  5. 2023Multilingual Jailbreak Challenges in Large Language Modelscited by 222
  6. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 189
  7. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 186
  8. 2023Can LLM-Generated Misinformation Be Detected?cited by 173
  9. 2023CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languagescited by 172
  10. 2023Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Promptingcited by 163
  11. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
  12. 2023M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Modelscited by 109
38 more
  1. 2023SeaLLMs -- Large Language Models for Southeast Asiacited by 104
  2. 2023LLMs for Knowledge Graph Construction and Reasoning: Recent Capabilities and Future Opportunitiescited by 97
  3. 2023SeaEval for Multilingual Foundation Models: From Cross-Lingual Alignment to Cultural Reasoningcited by 94
  4. 2023GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformercited by 84
  5. 2024A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendationscited by 84
  6. 2023GPTEval: A Survey on Assessments of ChatGPT and GPT-4cited by 73
  7. 2023Large Language Models and Knowledge Graphs: Opportunities and Challengescited by 71
  8. 2023MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities, Models and Taskscited by 57
  9. 2023Prompt Engineering for Healthcare: Methodologies and Applicationscited by 52
  10. 2023All Languages Matter: On the Multilingual Safety of Large Language Modelscited by 51
  11. 2024Understanding the Capabilities and Limitations of Large Language Models for Cultural Commonsensecited by 50
  12. 2023ChatGPT vs Human-authored Text: Insights into Controllable Text Summarization and Sentence Style Transfercited by 49
  13. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 49
  14. 2023Detecting and Mitigating Hallucinations in Multilingual Summarisationcited by 48
  15. 2024A Comprehensive Evaluation of Quantization Strategies for Large Language Modelscited by 47
  16. 2023AfroBench: How Good are Large Language Models on African Languages?cited by 43
  17. 2024IrokoBench: A New Benchmark for African Languages in the Age of Large Language Modelscited by 43
  18. 2023Small Models are Valuable Plug-ins for Large Language Modelscited by 42
  19. 2023Turning English-centric LLMs Into Polyglots: How Much Multilinguality Is Needed?cited by 39
  20. 2023GPTAraEval: A Comprehensive Evaluation of ChatGPT on Arabic NLPcited by 37
  21. 2024A Cross-Language Investigation into Jailbreak Attacks in Large Language Modelscited by 37
  22. 2024Gender Bias in Large Language Models across Multiple Languagescited by 35
  23. 2024RTP-LX: Can LLMs Evaluate Toxicity in Multilingual Scenarios?cited by 34
  24. 2024MBBQ: A Dataset for Cross-Lingual Comparison of Stereotypes in Generative LLMscited by 32
  25. 2024On the Challenges and Opportunities in Generative AIcited by 31
  26. 2024Is Translation All You Need? A Study on Solving Multilingual Tasks with Large Language Modelscited by 31
  27. 2023A Fast Optimization View: Reformulating Single Layer Attention in LLM Based on Tensor and SVM Trick, and Solving It in Matrix Multiplication Timecited by 30
  28. 2024NativQA: Multilingual Culturally-Aligned Natural Query for LLMscited by 30
  29. 2025Show Me the Work: Fact-Checkers' Requirements for Explainable Automated Fact-Checkingcited by 29
  30. 2024MaLA-500: Massive Language Adaptation of Large Language Modelscited by 28
  31. 2024Evaluating the Elementary Multilingual Capabilities of Large Language Models with MultiQcited by 27
  32. 2024Multilingual Pretraining and Instruction Tuning Improve Cross-Lingual Knowledge Alignment, But Only Shallowlycited by 27
  33. 2024MEXA: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignmentcited by 25
  34. 2023Using Large Language Models for Zero-Shot Natural Language Generation from Knowledge Graphscited by 24
  35. 2024Bridging the Language Gaps in Large Language Models with Inference-Time Cross-Lingual Interventioncited by 24
  36. 2023BenLLMEval: A Comprehensive Evaluation into the Potentials and Pitfalls of Large Language Models on Bengali NLPcited by 23
  37. 2024Under the Surface: Tracking the Artifactuality of LLM-Generated Datacited by 23
  38. 2024Multilingual Blending: LLM Safety Alignment Evaluation with Language Mixturecited by 22

50 most cited of 203 All 203 (JSON)