mlsanityData to 7 Oct 2026

Don't Trust ChatGPT when Your Question is not in English: A Study of Multilingual Abilities and Types of LLMs

arXiv 2305.16339 ↗First posted 24 May 2023cs.CL cs.AIcited by 103

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have demonstrated exceptional natural language understanding abilities and have excelled in a variety of natural language processing (NLP)tasks in recent years. Despite the fact that most LLMs are trained predominantly in English, multiple studies have demonstrated their comparative performance in many other languages. However, fundamental questions persist regarding how LLMs acquire their multi-lingual abilities and how performance varies across different languages. These inquiries are crucial for the study of LLMs since users and researchers often come from diverse language backgrounds, potentially influencing their utilization and interpretation of LLMs' results. In this work, we propose a systematic way of qualifying the performance disparities of LLMs under multilingual settings. We investigate the phenomenon of across-language generalizations in LLMs, wherein insufficient multi-lingual training data leads to advanced multi-lingual capabilities. To accomplish this, we employ a novel back-translation-based prompting method. The results show that GPT exhibits highly translating-like behaviour in multilingual settings.

Key facts. Cited by 103 papers in the map (data through 7 Oct 2026). Builds on 15 of its 32 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

15 of 32 references.

References · 15

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  5. 2021Editing Factual Knowledge in Language Modelscited by 548
  6. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 486
  7. 2020Deep Learning for Text Style Transfer: A Surveycited by 161
  8. 2021Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Surveycited by 130
  9. 2022A Survey of Deep Learning for Mathematical Reasoningcited by 124
  10. 2020Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queriescited by 107
  11. 2014Question Answering with Subgraph Embeddingscited by 56
  12. 2023LLM-powered Data Augmentation for Enhanced Cross-lingual Performancecited by 51
3 more
  1. 2022Beyond Counting Datasets: A Survey of Multilingual Dataset Construction and Necessary Resourcescited by 25
  2. 2022Modern Question Answering Datasets and Benchmarks: A Surveycited by 23
  3. 2019One Homonym per Translationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 39; 2025: 44; 2026: 18.

103 papers.

Sort:

By area:

  1. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 275
  2. 2023Do Multilingual Language Models Think Better in English?cited by 87
  3. 2024MAP-Neo: Highly Capable and Transparent Bilingual Large Language Model Seriescited by 67
  4. 2024How Well Do LLMs Represent Values Across Cultures? Empirical Analysis of LLM Responses Based on Hofstede Cultural Dimensionscited by 51
  5. 2025Lost in Multilinguality: Dissecting Cross-lingual Factual Inconsistency in Transformer Language Modelscited by 43
  6. 2024Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languagescited by 38
  7. 2024Evaluating Cultural Adaptability of a Large Language Model via Simulation of Synthetic Personascited by 36
  8. 2024KorNAT: LLM Alignment Benchmark for Korean Social Values and Common Knowledgecited by 33
  9. 2024Is Translation All You Need? A Study on Solving Multilingual Tasks with Large Language Modelscited by 31
  10. 2024Counting Ability of Large Language Models and Impact of Tokenizationcited by 29
  11. 2025Why Prompt Design Matters and Works: A Complexity Analysis of Prompt Search Space in LLMscited by 29
  12. 2024Evaluating the Elementary Multilingual Capabilities of Large Language Models with MultiQcited by 27
38 more
  1. 2024Multilingual Pretraining and Instruction Tuning Improve Cross-Lingual Knowledge Alignment, But Only Shallowlycited by 27
  2. 2024Exploring Multilingual Concepts of Human Value in Large Language Models: Is Value Alignment Consistent, Transferable and Controllable across Languages?cited by 23
  3. 2024PreAlign: Boosting Cross-Lingual Transfer by Early Establishment of Multilingual Alignmentcited by 20
  4. 2025Localized Cultural Knowledge is Conserved and Controllable in Large Language Modelscited by 19
  5. 2025TimeSeriesScientist: A General-Purpose AI Agent for Time Series Analysiscited by 18
  6. 2024Cross-Modal Consistency in Multimodal Large Language Modelscited by 17
  7. 2024Autoregressive + Chain of Thought = Recurrent: Recurrence's Role in Language Models' Computability and a Revisit of Recurrent Transformercited by 16
  8. 2024Teaching LLMs to Abstain across Languages via Multilingual Feedbackcited by 14
  9. 2025Hallucination to Truth: A Review of Fact-Checking and Factuality Evaluation in Large Language Modelscited by 13
  10. 2024A Principled Framework for Evaluating on Typologically Diverse Languagescited by 12
  11. 2025Multi2: Multi-Agent Test-Time Scalable Framework for Multi-Document Processingcited by 12
  12. 2024Getting More from Less: Large Language Models are Good Spontaneous Multilingual Learnerscited by 11
  13. 2024Creating a Lens of Chinese Culture: A Multimodal Dataset for Chinese Pun Rebus Art Understandingcited by 11
  14. 2024CROPE: Evaluating In-Context Adaptation of Vision and Language Models to Culture-Specific Conceptscited by 11
  15. 2024mCSQA: Multilingual Commonsense Reasoning Dataset with Unified Creation Strategy by Language Models and Humanscited by 10
  16. 2024Towards More Trustworthy and Interpretable LLMs for Code through Syntax-Grounded Explanationscited by 10
  17. 2025Soteria: Language-Specific Functional Parameter Steering for Multilingual Safety Alignmentcited by 10
  18. 2025CausalAbstain: Enhancing Multilingual LLMs with Causal Reasoning for Trustworthy Abstentioncited by 10
  19. 2026Align Once, Benefit Multilingually: Enforcing Multilingual Consistency for LLM Safety Alignmentcited by 9
  20. 2023Playing with words: Comparing the vocabulary and lexical diversity of ChatGPT and humanscited by 8
  21. 2025Large Language Models, Knowledge Graphs and Search Engines: A Crossroads for Answering Users' Questionscited by 8
  22. 2025SinLlama -- A Large Language Model for Sinhalacited by 8
  23. 2024Multilingual Hallucination Gaps in Large Language Modelscited by 7
  24. 2024The Roles of English in Evaluating Multilingual Language Modelscited by 7
  25. 2024LlamaTurk: Adapting Open-Source Generative Large Language Models for Low-Resource Languagecited by 6
  26. 2025DeepSeek-R1 Outperforms Gemini 2.0 Pro, OpenAI o1, and o3-mini in Bilingual Complex Ophthalmology Reasoningcited by 6
  27. 2025Has the Creativity of Large-Language Models peaked? An analysis of inter- and intra-LLM variabilitycited by 6
  28. 2025Characterizing Bias: Benchmarking Large Language Models in Simplified versus Traditional Chinesecited by 6
  29. 2024Native Design Bias: Studying the Impact of English Nativeness on Language Model Performancecited by 5
  30. 2024Chain-of-Translation Prompting (CoTR): A Novel Prompting Technique for Low Resource Languagescited by 5
  31. 2024Supervised Chain of Thoughtcited by 5
  32. 2025Retrieval is Not Enough: Enhancing RAG Reasoning through Test-Time Critique and Optimizationcited by 5
  33. 2025It's All About In-Context Learning! Teaching Extremely Low-Resource Languages to LLMscited by 5
  34. 2025Document Intelligence in the Era of Large Language Models: A Surveycited by 5
  35. 2024Unlocking Parameter-Efficient Fine-Tuning for Low-Resource Language Translationcited by 4
  36. 2024Why We Build Local Large Language Models: An Observational Analysis from 35 Japanese and Multilingual LLMscited by 4
  37. 2024KnowCoder-X: Boosting Multilingual Information Extraction via Codecited by 3
  38. 2024Seed-Free Synthetic Data Generation Framework for Instruction-Tuning LLMs: A Case Study in Thaicited by 3

50 most cited of 103 All 103 (JSON)