mlsanityData to 13 Oct 2026

Understanding and Mitigating Language Confusion in LLMs

arXiv 2406.20052 ↗First posted 28 Jun 2024cs.CLcited by 75

Not on the map. Why?

Abstract (arXiv)

We investigate a surprising limitation of LLMs: their inability to consistently generate text in a user's desired language. We create the Language Confusion Benchmark (LCB) to evaluate such failures, covering 15 typologically diverse languages with existing and newly-created English and multilingual prompts. We evaluate a range of LLMs on monolingual and cross-lingual generation reflecting practical use cases, finding that Llama Instruct and Mistral models exhibit high degrees of language confusion and even the strongest models fail to consistently respond in the correct language. We observe that base and English-centric instruct models are more prone to language confusion, which is aggravated by complex prompts and high sampling temperatures. We find that language confusion can be partially mitigated via few-shot prompting, multilingual SFT and preference tuning. We release our language confusion benchmark, which serves as a first layer of efficient, scalable multilingual evaluation at https://github.com/for-ai/language-confusion.

Key facts. Cited by 75 papers in the map (data through 13 Oct 2026). Builds on 36 of its 52 references that are in the map. First posted on arXiv 28 Jun 2024.

Builds on

36 of 52 references.

References · 36

  1. 2020Language Models are Few-Shot Learnerscited by 32,320
  2. 2023GPT-4 Technical Reportcited by 18,278
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,979
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,986
  5. 2019The Curious Case of Neural Text Degenerationcited by 2,893
  6. 2016Bag of Tricks for Efficient Text Classificationcited by 872
  7. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 792
  8. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 741
  9. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 292
  10. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  11. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 192
  12. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 190
24 more
  1. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 189
  2. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 174
  3. 2023Hallucinations in Large Multilingual Translation Modelscited by 129
  4. 2024Aya 23: Open Weight Releases to Further Multilingual Progresscited by 123
  5. 2020GLUECoS : An Evaluation Benchmark for Code-Switched NLPcited by 101
  6. 2024LLaMA Beyond English: An Empirical Study on Language Capability Transfercited by 94
  7. 2022Language Contamination Helps Explain the Cross-lingual Capabilities of English Pretrained Modelscited by 92
  8. 2023Human Feedback is not Gold Standardcited by 82
  9. 2023BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfercited by 68
  10. 2023Multilingual Large Language Models Are Not (Yet) Code-Switcherscited by 65
  11. 2022Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generationcited by 62
  12. 2023Monolingual or Multilingual Instruction Tuning: Which Makes a Better Alpacacited by 60
  13. 2023A Survey of Code-switching: Linguistic and Social Perspectives for Language Technologiescited by 59
  14. 2023Prompting Multilingual Large Language Models to Generate Code-Mixed Texts: The Case of South East Asian Languagescited by 50
  15. 2023Mitigating Hallucinations and Off-target Machine Translation with Source-Contrastive and Language-Contrastive Decodingcited by 42
  16. 2023Turning English-centric LLMs Into Polyglots: How Much Multilinguality Is Needed?cited by 39
  17. 2022The Decades Progress on Code-Switching Research in NLP: A Systematic Survey on Trends and Challengescited by 35
  18. 2022Geographic and Geopolitical Biases of Language Modelscited by 29
  19. 2024Evaluating the Elementary Multilingual Capabilities of Large Language Models with MultiQcited by 27
  20. 2023mmT5: Modular Multilingual Pre-Training Solves Source Language Hallucinationscited by 24
  21. 20243D-Properties: Identifying Challenges in DPO and Charting a Path Forwardcited by 24
  22. 2023Why Does Zero-Shot Cross-Lingual Generation Fail? An Explanation and a Solutioncited by 22
  23. 2023A Material Lens on Coloniality in NLPcited by 19
  24. 2024Key ingredients for effective zero-shot cross-lingual knowledge transfer in generative taskscited by 11

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 17; 2025: 32; 2026: 26.

75 papers.

Sort:

By area:

  1. 2025Do Multilingual LLMs Think In English?cited by 78
  2. 2024LLMmap: Fingerprinting For Large Language Modelscited by 58
  3. 2024How Does Quantization Affect Multilingual LLMs?cited by 42
  4. 2024MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Modelscited by 34
  5. 2024Constructing Domain-Specific Evaluation Sets for LLM-as-a-judgecited by 30
  6. 2025Language Mixing in Reasoning Language Models: Patterns, Impact, and Internal Causescited by 21
  7. 2026Tiny Aya: Bridging Scale and Multilingual Depthcited by 15
  8. 2025Standard Benchmarks Fail -- Auditing LLM Agents in Finance Must Prioritize Riskcited by 13
  9. 2025The Impact of Language Mixing on Bilingual LLM Reasoningcited by 12
  10. 2024AL-QASIDA: Analyzing LLM Quality and Accuracy Systematically in Dialectal Arabiccited by 11
  11. 2025Logits DeConfusion with CLIP for Few-Shot Learningcited by 11
  12. 2025Cross-lingual Collapse: How Language-Centric Foundation Models Shape Reasoning in Large Language Modelscited by 11
38 more
  1. 2024Against All Odds: Overcoming Typology, Script, and Language Confusion in Multilingual Embedding Inversion Attackscited by 10
  2. 2024Multi-OphthaLingua: A Multilingual Benchmark for Assessing and Debiasing LLM Ophthalmological QA in LMICscited by 10
  3. 2025Culture is Everywhere: A Call for Intentionally Cultural Evaluationcited by 10
  4. 2025Conditions for Catastrophic Forgetting in Multilingual Translationcited by 10
  5. 2024Native vs Non-Native Language Prompting: A Comparative Analysiscited by 9
  6. 2025The Translation Barrier Hypothesis: Multilingual Generation with Large Language Models Suffers from Implicit Translation Failurecited by 9
  7. 2025Mechanistic Understanding and Mitigation of Language Confusion in English-Centric Large Language Modelscited by 8
  8. 2024Understanding Likelihood Over-optimisation in Direct Alignment Algorithmscited by 7
  9. 2024Large Language Models are Easily Confused: A Quantitative Metric, Security Implications and Typological Analysiscited by 7
  10. 2024BgGPT 1.0: Extending English-centric LLMs to other languagescited by 7
  11. 2024LlamaLens: Specialized Multilingual LLM for Analyzing News and Social Media Contentcited by 6
  12. 2025Evalita-LLM: Benchmarking Large Language Models on Italiancited by 6
  13. 2025A Post-trainer's Guide to Multilingual Training Data: Uncovering Cross-lingual Transfer Dynamicscited by 6
  14. 2025ReCoVeR the Target Language: Language Steering without Sacrificing Task Performancecited by 6
  15. 2024From English-Centric to Effective Bilingual: LLMs with Custom Tokenizers for Underrepresented Languagescited by 5
  16. 2024Multilingual Pretraining Using a Large Corpus Machine-Translated from a Single Source Languagecited by 5
  17. 2024Cross-Lingual Transfer of Debiasing and Detoxification in Multilingual LLMs: An Extensive Investigationcited by 5
  18. 2025Is Small Language Model the Silver Bullet to Low-Resource Languages Machine Translation?cited by 5
  19. 2025The Art of Asking: Multilingual Prompt Optimization for Synthetic Datacited by 5
  20. 2025MiLiC-Eval: Benchmarking Multilingual LLMs for China's Minority Languagescited by 4
  21. 2025Language Surgery in Multilingual Large Language Modelscited by 4
  22. 2025UPDESH: Synthesizing Grounded Instruction Tuning Data for 13 Indic Languagescited by 4
  23. 2025One Model, Many Morals: Uncovering Cross-Linguistic Misalignments in Computational Moral Reasoningcited by 4
  24. 2025Beyond Monolingual Assumptions: A Survey of Code-Switched NLP in the Era of Large Language Models across Modalitiescited by 4
  25. 2025Uncovering the Potential Risks in Unlearning: Danger of English-only Unlearning in Multilingual LLMscited by 4
  26. 2025Controlling Language Confusion in Multilingual LLMscited by 3
  27. 2025MENLO: From Preferences to Proficiency -- Evaluating and Modeling Native-like Quality Across 47 Languagescited by 3
  28. 2026The Script Tax: Measuring Tokenization-Driven Efficiency and Latency Disparities in Multilingual Language Modelscited by 3
  29. 2025Exploring the Feasibility of Multilingual Grammatical Error Correction with a Single LLM up to 9B parameters: A Comparative Study of 17 Modelscited by 2
  30. 2025Cross-Lingual Optimization for Language Transfer in Large Language Modelscited by 2
  31. 2025Treasure Hunt: Real-time Targeting of the Long Tail using Training-Time Markerscited by 2
  32. 2025Smoothie-Qwen: Post-Hoc Smoothing to Reduce Language Bias in Multilingual LLMscited by 2
  33. 2026OLA: Output Language Alignment in Code-Switched LLM Interactionscited by 2
  34. 2025XL-Suite: Cross-Lingual Synthetic Training and Evaluation Data for Open-Ended Generationcited by 1
  35. 2025MateInfoUB: A Real-World Benchmark for Testing LLMs in Competitive, Multilingual, and Multimodal Educational Taskscited by 1
  36. 2025SiniticMTError: A Machine Translation Dataset with Error Annotations for Sinitic Languagescited by 1
  37. 2025Language Confusion Gate: Language-Aware Decoding Through Model Self-Distillationcited by 1
  38. 2025AI Text Detectors and the Misclassification of Slightly Polished Arabic Textcited by 1

50 most cited of 75 All 75 (JSON)