mlsanityData to 14 Oct 2026

Multilingual Large Language Models and Curse of Multilinguality

arXiv 2406.10602 ↗First posted 15 Jun 2024cs.CL cs.CYcited by 13

Not on the map. Why?

Abstract (arXiv)

Multilingual Large Language Models (LLMs) have gained large popularity among Natural Language Processing (NLP) researchers and practitioners. These models, trained on huge datasets, show proficiency across various languages and demonstrate effectiveness in numerous downstream tasks. This paper navigates the landscape of multilingual LLMs, providing an introductory overview of their technical aspects. It explains underlying architectures, objective functions, pre-training data sources, and tokenization methods. This work explores the unique features of different model types: encoder-only (mBERT, XLM-R), decoder-only (XGLM, PALM, BLOOM, GPT-3), and encoder-decoder models (mT5, mBART). Additionally, it addresses one of the significant limitations of multilingual LLMs - the curse of multilinguality - and discusses current attempts to overcome it.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 28 of its 37 references that are in the map. First posted on arXiv 15 Jun 2024.

Builds on

28 of 37 references.

References · 28

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  7. 2016Layer Normalizationcited by 5,669
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  10. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  11. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  12. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
16 more
  1. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  2. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  3. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
  4. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  5. 2019How multilingual is Multilingual BERT?cited by 645
  6. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
  7. 2017Convolutional Sequence to Sequence Learningcited by 552
  8. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 548
  9. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 445
  10. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  11. 2019Massively Multilingual Neural Machine Translationcited by 246
  12. 2020Are All Languages Created Equal in Multilingual BERT?cited by 207
  13. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 146
  14. 2022Towards a Cleaner Document-Oriented Multilingual Crawled Corpuscited by 143
  15. 2020When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Modelscited by 131
  16. 2021Subword Mapping and Anchoring across Languagescited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 9; 2026: 2.

13 papers.

Sort:

By area:

  1. 2025PolyGuard: A Multilingual Safety Moderation Tool for 17 Languagescited by 59
  2. 2025TreeReview: A Dynamic Tree of Questions Framework for Deep and Efficient LLM-based Scientific Peer Reviewcited by 31
  3. 2025Learn Globally, Speak Locally: Bridging the Gaps in Multilingual Reasoningcited by 24
  4. 2025Localized Cultural Knowledge is Conserved and Controllable in Large Language Modelscited by 20
  5. 2024Multilingual Large Language Models: A Systematic Surveycited by 18
  6. 2025Find Your Optimal Teacher: Personalized Data Synthesis via Router-Guided Multi-Teacher Distillationcited by 15
  7. 2024FuxiTranyu: A Multilingual Large Language Model Trained with Balanced Datacited by 10
  8. 2025Soteria: Language-Specific Functional Parameter Steering for Multilingual Safety Alignmentcited by 10
  9. 2025GuiLoMo: Allocating Expert Number and Rank for LoRA-MoE via Bilevel Optimization with GuidedSelection Vectorscited by 6
  10. 2025Language Specific Knowledge: Do Models Know Better in X than in English?cited by 4
  11. 2025Safer in Translation? Presupposition Robustness in Indic Languagescited by 0
  12. 2026MoLGE: Mixture of Language Group Experts for Efficient Scaling of Massively Multilingual Speech Recognitioncited by 0
1 more
  1. 2026There is No Theoretical Curse of Multilinguality For Embedding Space Structurecited by 0