mlsanityData to 7 Oct 2026

Breaking the Curse of Multilinguality with Cross-lingual Expert Language Models

arXiv 2401.10440 ↗First posted 19 Jan 2024cs.CLcited by 36

Not on the map. Why?

Abstract (arXiv)

Despite their popularity in non-English NLP, multilingual language models often underperform monolingual ones due to inter-language competition for model parameters. We propose Cross-lingual Expert Language Models (X-ELM), which mitigate this competition by independently training language models on subsets of the multilingual corpus. This process specializes X-ELMs to different languages while remaining effective as a multilingual ensemble. Our experiments show that when given the same compute budget, X-ELM outperforms jointly trained multilingual models across all considered languages and that these gains transfer to downstream tasks. X-ELM provides additional benefits over performance improvements: new experts can be iteratively added, adapting X-ELM to new languages without catastrophic forgetting. Furthermore, training is asynchronous, reducing the hardware requirements for multilingual training and democratizing multilingual modeling.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Builds on 31 of its 46 references that are in the map. First posted on arXiv 19 Jan 2024.

Builds on

31 of 46 references.

References · 31

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  3. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  4. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  5. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  6. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  7. 2019Rigging the Lottery: Making All Tickets Winnerscited by 491
  8. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 443
  9. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 413
  10. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  11. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 262
  12. 2019Sparse Networks from Scratch: Faster Training without Losing Performancecited by 210
19 more
  1. 2020Are All Languages Created Equal in Multilingual BERT?cited by 206
  2. 2019Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterizationcited by 187
  3. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 187
  4. 2023UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretrainingcited by 147
  5. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 145
  6. 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 130
  7. 2021Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inferencecited by 96
  8. 2022Language Contamination Helps Explain the Cross-lingual Capabilities of English Pretrained Modelscited by 90
  9. 2023XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Modelscited by 88
  10. 2021Larger-Scale Transformers for Multilingual Masked Language Modelingcited by 86
  11. 2023Exploring the Benefits of Training Expert Language Models over Instruction Tuningcited by 80
  12. 2019Learning and Evaluating General Linguistic Intelligencecited by 78
  13. 2021XLM-E: Cross-lingual Language Model Pre-training via ELECTRAcited by 72
  14. 2023Scaling Expert Language Models with Unsupervised Domain Discoverycited by 59
  15. 2021How to Adapt Your Pretrained Multilingual Model to 1600 Languagescited by 54
  16. 2022Analyzing the Mono- and Cross-Lingual Pretraining Dynamics of Multilingual Language Modelscited by 32
  17. 2023Overcoming Catastrophic Forgetting in Massively Multilingual Continual Learningcited by 27
  18. 2022Phylogeny-Inspired Adaptation of Multilingual Models to New Languagescited by 24
  19. 2021PARADISE: Exploiting Parallel Data for Multilingual Sequence-to-Sequence Pretrainingcited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 13; 2026: 10.

36 papers.

Sort:

By area:

  1. 2024Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabulariescited by 113
  2. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 49
  3. 2024Parrot: Multilingual Visual Instruction Tuningcited by 30
  4. 2024Goldfish: Monolingual Language Models for 350 Languagescited by 27
  5. 2024IRCoder: Intermediate Representations Make Language Models Robust Multilingual Code Generatorscited by 21
  6. 2024Multilingual Large Language Models: A Systematic Surveycited by 18
  7. 2024MoE-LPR: Multilingual Extension of Large Language Models through Mixture-of-Experts with Language Priors Routingcited by 16
  8. 2025Reasoning Beyond Limits: Advances and Open Problems for LLMscited by 16
  9. 2024SambaLingo: Teaching Large Language Models New Languagescited by 11
  10. 2025Revisiting Multilingual Data Mixtures in Language Model Pretrainingcited by 10
  11. 2025Bring Your Own Knowledge: A Survey of Methods for LLM Knowledge Expansioncited by 8
  12. 2025Trustworthy Machine Learning via Memorization and the Granular Long-Tail: A Survey on Interactions, Tradeoffs, and Beyondcited by 6
24 more
  1. 2025An Interdisciplinary Approach to Human-Centered Machine Translationcited by 6
  2. 2024M2QA: Multi-domain Multilingual Question Answeringcited by 5
  3. 2024Lens: Rethinking Multilingual Enhancement for Large Language Modelscited by 5
  4. 2024From English-Centric to Effective Bilingual: LLMs with Custom Tokenizers for Underrepresented Languagescited by 5
  5. 2026Cross-Lingual Sentiment Misalignment: Auditing Multilingual Language Models for Inversion Risk, Dialectal Representation, and Affective Stabilitycited by 5
  6. 2025Mitigating Catastrophic Forgetting in Target Language Adaptation of LLMs via Source-Shielded Updatescited by 4
  7. 2025Kuwain 1.5B: An Arabic SLM via Language Injectioncited by 3
  8. 2025IMPACT: Inflectional Morphology Probes Across Complex Typologiescited by 3
  9. 2025Grow Up and Merge: Scaling Strategies for Efficient Language Adaptationcited by 3
  10. 2026"UberWeb: Insights from Multilingual Curation for a 20-Trillion-Token Datasetcited by 3
  11. 2024Modular Sentence Encoders: Separating Language Specialization from Cross-Lingual Alignmentcited by 2
  12. 2025Assessing the Role of Data Quality in Training Bilingual Language Modelscited by 2
  13. 2026Soro: A Lightweight Foundation Model and Chatbot for Tajikcited by 2
  14. 2025TabiBERT: A Large-Scale ModernBERT Foundation Model and A Unified Benchmark for Turkishcited by 1
  15. 2026Multilinguality at the Edge: Developing Language Models for the Global Southcited by 1
  16. 2024Exploring Continual Fine-Tuning for Enhancing Language Ability in Large Language Modelcited by 0
  17. 2025How to Tune a Multilingual Encoder Model for Germanic Languages: A Study of PEFT, Full Fine-Tuning, and Language Adapterscited by 0
  18. 2025Discriminating Form and Meaning in Multilingual Models with Minimal-Pair ABX Taskscited by 0
  19. 2026Capacity Constraints and the Multilingual Penalty for Lexical Disambiguationcited by 0
  20. 2026Is Textual Similarity Invariant under Machine Translation? Evidence Based on the Political Manifesto Corpuscited by 0
  21. 2026Specializing Without Forgetting: Analyzing Knowledge Preservation in Multilingual Model Adaptationcited by 0
  22. 2026Modular Monolingual Adaptation using Pretrained Language Modelscited by 0
  23. 2026MET: Theory-Grounded and Culture-Aware Multilingual Moral Reasoningcited by 0
  24. 2026Cross-Lingual Transfer for Machine Translation in Turkic Languagescited by 0