mlsanityData to 7 Oct 2026

When Is Multilinguality a Curse? Language Modeling for 250 High- and Low-Resource Languages

arXiv 2311.09205 ↗First posted 15 Nov 2023cs.CLcited by 32

Not on the map. Why?

Abstract (arXiv)

Multilingual language models are widely used to extend NLP systems to low-resource languages. However, concrete evidence for the effects of multilinguality on language modeling performance in individual languages remains scarce. Here, we pre-train over 10,000 monolingual and multilingual language models for over 250 languages, including multiple language families that are under-studied in NLP. We assess how language modeling performance in each language varies as a function of (1) monolingual dataset size, (2) added multilingual dataset size, (3) linguistic similarity of the added languages, and (4) model size (up to 45M parameters). We find that in moderation, adding multilingual data improves low-resource language modeling performance, similar to increasing low-resource dataset sizes by up to 33%. Improvements depend on the syntactic similarity of the added multilingual data, with marginal additional effects of vocabulary overlap. However, high-resource languages consistently perform worse in multilingual pre-training scenarios. As dataset sizes increase, adding multilingual data begins to hurt performance for both low-resource and high-resource languages, likely due to limited model capacity (the "curse of multilinguality"). These results suggest that massively multilingual pre-training may not be optimal for any languages involved, but that more targeted models can significantly improve performance.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 37 of its 78 references that are in the map. First posted on arXiv 15 Nov 2023.

Builds on

37 of 78 references.

References · 37

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  5. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  7. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  8. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  9. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  10. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  11. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  12. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
25 more
  1. 2019How multilingual is Multilingual BERT?cited by 642
  2. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 313
  3. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 300
  4. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  5. 2020Are All Languages Created Equal in Multilingual BERT?cited by 206
  6. 2019Cross-Lingual Ability of Multilingual BERT: An Empirical Studycited by 182
  7. 2019Emerging Cross-lingual Structure in Pretrained Language Modelscited by 180
  8. 2021MasakhaNER: Named Entity Recognition for African Languagescited by 155
  9. 2022Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuningcited by 149
  10. 2020The Tatoeba Translation Challenge -- Realistic Data Sets for Low Resource and Multilingual MTcited by 127
  11. 2021Systematic Inequalities in Language Technology Performance across the World's Languagescited by 118
  12. 2023Glot500: Scaling Multilingual Corpora and Language Models to 500 Languagescited by 116
  13. 2021Language Models are Few-shot Multilingual Learnerscited by 92
  14. 2022Training Trajectories of Language Models Across Scalescited by 80
  15. 2022NusaX: Multilingual Parallel Sentiment Dataset for 10 Indonesian Local Languagescited by 67
  16. 2021AmericasNLI: Evaluating Zero-shot Natural Language Understanding of Pretrained Multilingual Models in Truly Low-resource Languagescited by 50
  17. 2021Word Acquisition in Neural Language Modelscited by 41
  18. 2022Analyzing the Mono- and Cross-Lingual Pretraining Dynamics of Multilingual Language Modelscited by 32
  19. 2023Zero-shot cross-lingual transfer language selection using linguistic similaritycited by 28
  20. 2022Match the Script, Adapt if Multilingual: Analyzing the Effect of Multilingual Pretraining on Cross-lingual Transferabilitycited by 26
  21. 2020WikiBERT models: deep transfer learning for many languagescited by 20
  22. 2020ChrEn: Cherokee-English Machine Translation for Endangered Language Revitalizationcited by 20
  23. 2022On the Calibration of Massively Multilingual Language Modelscited by 18
  24. 2022Cross-Lingual Ability of Multilingual Masked Language Models: A Study of Language Structurecited by 17
  25. 2022The Makerere Radio Speech Corpus: A Luganda Radio Corpus for Automatic Speech Recognitioncited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 16; 2026: 10.

32 papers.

Sort:

By area:

  1. 2025MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairscited by 65
  2. 2025Enhancing Multilingual LLM Pretraining with Model-Based Data Selectioncited by 33
  3. 2024Parrot: Multilingual Visual Instruction Tuningcited by 30
  4. 2024BertaQA: How Much Do Language Models Know About Local Culture?cited by 23
  5. 2025Learn Globally, Speak Locally: Bridging the Gaps in Multilingual Reasoningcited by 23
  6. 2025Localized Cultural Knowledge is Conserved and Controllable in Large Language Modelscited by 19
  7. 2025ATLAS: Adaptive Transfer Scaling Laws for Multilingual Pretraining, Finetuning, and Decoding the Curse of Multilingualitycited by 19
  8. 2025Rethinking Multilingual Continual Pretraining: Data Mixing for Adapting LLMs Across Languages and Resourcescited by 10
  9. 2025Revisiting Multilingual Data Mixtures in Language Model Pretrainingcited by 10
  10. 2025TALES: A Taxonomy and Analysis of Cultural Representations in LLM-generated Storiescited by 10
  11. 2024Evaluating and Mitigating Linguistic Discrimination in Large Language Modelscited by 8
  12. 2025Trustworthy Machine Learning via Memorization and the Granular Long-Tail: A Survey on Interactions, Tradeoffs, and Beyondcited by 6
20 more
  1. 2024Foundation Models for Low-Resource Language Education (Vision Paper)cited by 5
  2. 2024DEPT: Decoupled Embeddings for Pre-training Language Modelscited by 4
  3. 2024Why We Build Local Large Language Models: An Observational Analysis from 35 Japanese and Multilingual LLMscited by 4
  4. 2025Toward a Theory of Generalizability in LLM Mechanistic Interpretability Researchcited by 4
  5. 2025The Evolution of Reranking Models in Information Retrieval: From Heuristic Methods to Large Language Modelscited by 3
  6. 2026"UberWeb: Insights from Multilingual Curation for a 20-Trillion-Token Datasetcited by 3
  7. 2026One Model to Translate Them All? A Journey to Mount Doom for Multilingual Model Mergingcited by 2
  8. 2026Multilingual Unlearning in LLMs: Transfer, Dynamics, and Reversibilitycited by 2
  9. 2025TabiBERT: A Large-Scale ModernBERT Foundation Model and A Unified Benchmark for Turkishcited by 1
  10. 2026The Fragility of Chain-of-Thought Monitoring Across Typologically Diverse Languagescited by 1
  11. 2025Exploring Polyglot Harmony: On Multilingual Data Allocation for Large Language Models Pretrainingcited by 0
  12. 2025Safer in Translation? Presupposition Robustness in Indic Languagescited by 0
  13. 2025Persian-Phi: Efficient Cross-Lingual Adaptation of Compact LLMs via Curriculum Learningcited by 0
  14. 2025Gamayun's Path to Multilingual Mastery: Cost-Efficient Training of a 1.5B-Parameter LLMcited by 0
  15. 2026BYOL: Bring Your Own Language Into LLMscited by 0
  16. 2026Raising Bars, Not Parameters: LilMoo Compact Language Model for Hindicited by 0
  17. 2026English is Not All You Need: Systematically Exploring the Role of Multilinguality in LLM Post-Trainingcited by 0
  18. 2026Effects of Cross-lingual Evidence in Multilingual Medical Question Answeringcited by 0
  19. 2026Multilingual Knowledge Transfer under Data Constraints via Lexical Interventionscited by 0
  20. 2026A P=aninian Foundation for Indic Language Processingcited by 0