mlsanityData to 7 Oct 2026

Crosslingual Capabilities and Knowledge Barriers in Multilingual Large Language Models

arXiv 2406.16135 ↗First posted 23 Jun 2024cs.CL cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) are typically multilingual due to pretraining on diverse multilingual corpora. But can these models relate corresponding concepts across languages, i.e., be crosslingual? This study evaluates state-of-the-art LLMs on inherently crosslingual tasks. We observe that while these models show promising surface-level crosslingual abilities on machine translation and embedding space analyses, they struggle with deeper crosslingual knowledge transfer, revealing a crosslingual knowledge barrier in both general (MMLU benchmark) and domain-specific (Harry Potter quiz and TOFU benchmark) contexts. Since simple inference-time mitigation methods offer only limited improvement, we propose fine-tuning of LLMs on mixed-language data, which effectively reduces these gaps, even when using out-of-domain datasets like WikiText. Our findings suggest the need for explicit optimization to unlock the full crosslingual potential of LLMs. Our code is publicly available at https://github.com/google-research/crosslingual-knowledge-barriers.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 67 of its 78 references that are in the map. First posted on arXiv 23 Jun 2024.

Builds on

67 of 78 references.

References · 67

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  5. 2023Mistral 7Bcited by 3,065
  6. 2016Pointer Sentinel Mixture Modelscited by 2,992
  7. 2024Mixtral of Expertscited by 1,785
  8. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  9. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  10. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  11. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  12. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
38 more
  1. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  2. 2019How multilingual is Multilingual BERT?cited by 642
  3. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  4. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
  5. 2024TOFU: A Task of Fictitious Unlearning for LLMscited by 507
  6. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
  7. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  8. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 399
  9. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 396
  10. 2023Studying Large Language Model Generalization with Influence Functionscited by 311
  11. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
  12. 2019Compositionality decomposed: how do neural networks generalise?cited by 264
  13. 2020COGS: A Compositional Generalization Challenge Based on Semantic Interpretationcited by 260
  14. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 249
  15. 2019Simple, Scalable Adaptation for Neural Machine Translationcited by 239
  16. 2019CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEBcited by 177
  17. 2023Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Promptingcited by 163
  18. 2023Physically Grounded Vision-Language Models for Robotic Manipulationcited by 163
  19. 2023Multilingual Machine Translation with Large Language Models: Empirical Results and Analysiscited by 156
  20. 2022Towards a Cleaner Document-Oriented Multilingual Crawled Corpuscited by 143
  21. 2021Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Modelscited by 132
  22. 2024Aya 23: Open Weight Releases to Further Multilingual Progresscited by 121
  23. 2024Zamba: A Compact 7B SSM Hybrid Modelcited by 118
  24. 2020X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Modelscited by 117
  25. 2023Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Modelscited by 117
  26. 2023A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Modelscited by 88
  27. 2023Multilingual LLMs are Better Cross-lingual In-context Learners with Alignmentcited by 71
  28. 2024Premise Order Matters in Reasoning with Large Language Modelscited by 71
  29. 2023Extrapolating Large Language Models to Non-English by Aligning Languagescited by 69
  30. 2024Unintended Impacts of LLM Alignment on Global Representationcited by 66
  31. 2023Skill-Mix: a Flexible and Expandable Family of Evaluations for AI modelscited by 64
  32. 2022Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generationcited by 62
  33. 2019Code-Switching for Enhancing NMT with Pre-Specified Translationcited by 53
  34. 2023NEWTON: Are Large Language Models Capable of Physical Reasoning?cited by 45
  35. 2022TwHIN-BERT: A Socially-Enriched Pre-trained Language Model for Multilingual Tweet Representations at Twittercited by 44
  36. 2023Exploring Human-Like Translation Strategy with Large Language Modelscited by 44
  37. 2020Code-switching pre-training for neural machine translationcited by 43
  38. 2022Compositional Generalization in Unsupervised Compositional Representation Learning: A Study on Disentanglement and Emergent Languagecited by 42

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 3.

9 papers.

  1. 2025Large Language Models Hallucination: A Comprehensive Surveycited by 45
  2. 2025ECLeKTic: a Novel Challenge Set for Evaluation of Cross-Lingual Knowledge Transfercited by 28
  3. 2025Language-Specific Latent Process Hinders Cross-Lingual Performancecited by 20
  4. 2025MuBench: Assessment of Multilingual Capabilities of Large Language Models Across 61 Languagescited by 13
  5. 2025Rethinking Cross-lingual Gaps from a Statistical Viewpointcited by 4
  6. 2026Unveiling Language Routing Isolation in Multilingual MoE Models for Interpretable Subnetwork Adaptationcited by 4
  7. 2025Uncovering inequalities in new knowledge learning by large language models across different languagescited by 1
  8. 2026Do Language Models Reason Across Languages?cited by 1
  9. 2026Why Pretraining Fails to Share Cross-Lingual Knowledgecited by 0