mlsanityData to 7 Oct 2026

Lifting the Curse of Multilinguality by Pre-training Modular Transformers

arXiv 2205.06266 ↗First posted 12 May 2022cs.CLcited by 145

Not on the map. Why?

Abstract (arXiv)

Multilingual pre-trained models are known to suffer from the curse of multilinguality, which causes per-language performance to drop as they cover more languages. We address this issue by introducing language-specific modules, which allows us to grow the total capacity of the model, while keeping the total number of trainable parameters per language constant. In contrast with prior work that learns language-specific components post-hoc, we pre-train the modules of our Cross-lingual Modular (X-Mod) models from the start. Our experiments on natural language inference, named entity recognition and question answering show that our approach not only mitigates the negative interference between languages, but also enables positive transfer, resulting in improved monolingual and cross-lingual performance. Furthermore, our approach enables adding languages post-hoc with no measurable drop in performance, no longer limiting the model usage to the set of pre-trained languages.

Key facts. Cited by 145 papers in the map (data through 7 Oct 2026). Builds on 57 of its 80 references that are in the map. First posted on arXiv 12 May 2022.

Builds on

57 of 80 references.

References · 57

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  4. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  8. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  9. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  10. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  11. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  12. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
38 more
  1. 2019How multilingual is Multilingual BERT?cited by 642
  2. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  3. 2017Learning multiple visual domains with residual adapterscited by 553
  4. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  5. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  6. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 443
  7. 2020AdapterHub: A Framework for Adapting Transformerscited by 441
  8. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 339
  9. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 330
  10. 2020K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapterscited by 318
  11. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 313
  12. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
  13. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 278
  14. 2019Simple, Scalable Adaptation for Neural Machine Translationcited by 239
  15. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 229
  16. 2020AdapterDrop: On the Efficiency of Adapters in Transformerscited by 223
  17. 2020Are All Languages Created Equal in Multilingual BERT?cited by 206
  18. 2019Cross-Lingual Ability of Multilingual BERT: An Empirical Studycited by 182
  19. 2019Emerging Cross-lingual Structure in Pretrained Language Modelscited by 180
  20. 2021Charformer: Fast Character Transformers via Gradient-based Subword Tokenizationcited by 158
  21. 2021Composable Sparse Fine-Tuning for Cross-Lingual Transfercited by 144
  22. 2020When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Modelscited by 131
  23. 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 130
  24. 2020UNKs Everywhere: Adapting Multilingual Language Models to New Scriptscited by 113
  25. 2020Rethinking embedding coupling in pre-trained language modelscited by 103
  26. 2020Extending Multilingual BERT to Low-Resource Languagescited by 88
  27. 2016Learning to Compose Neural Networks for Question Answeringcited by 85
  28. 2020UDapter: Language Adaptation for Truly Universal Dependency Parsingcited by 77
  29. 2021What to Pre-Train on? Efficient Intermediate Task Selectioncited by 62
  30. 2020Common Sense or World Knowledge? Investigating Adapter-Based Knowledge Injection into Pretrained Transformerscited by 57
  31. 2020Improving Multilingual Models with Language-Clustered Vocabulariescited by 53
  32. 2021Lightweight Adapter Tuning for Multilingual Speech Translationcited by 53
  33. 2021AmericasNLI: Evaluating Zero-shot Natural Language Understanding of Pretrained Multilingual Models in Truly Low-resource Languagescited by 50
  34. 2019How to (Properly) Evaluate Cross-Lingual Word Embeddings: On Strong Baselines, Comparative Analyses, and Some Misconceptionscited by 48
  35. 2020From English To Foreign Languages: Transferring Pre-trained Language Modelscited by 37
  36. 2021Towards Continual Learning for Multilingual Machine Translation via Vocabulary Substitutioncited by 31
  37. 2021Efficient Test Time Adapter Ensembling for Low-resource Language Varietiescited by 26
  38. 2019Slice-based Learning: A Programming Model for Residual Learning in Critical Data Slicescited by 23

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 12; 2023: 47; 2024: 54; 2025: 17; 2026: 15.

145 papers.

Sort:

By area:

  1. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  2. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 292
  3. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 271
  4. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 187
  5. 2023SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialectscited by 149
  6. 2023UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretrainingcited by 147
  7. 2024Investigating Cultural Alignment of Large Language Modelscited by 140
  8. 2023Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluationcited by 134
  9. 2024Aya 23: Open Weight Releases to Further Multilingual Progresscited by 121
  10. 2023Glot500: Scaling Multilingual Corpora and Language Models to 500 Languagescited by 116
  11. 2022Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Datasetcited by 113
  12. 2023SLoRA: Federated Parameter Efficient Fine-Tuning of Language Modelscited by 92
38 more
  1. 2024KMMLU: Measuring Massive Multitask Language Understanding in Koreancited by 91
  2. 2022BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Promptingcited by 90
  3. 2023Soft Merging of Experts with Adaptive Routingcited by 88
  4. 2023Modular Deep Learningcited by 82
  5. 2023Exploring the Benefits of Training Expert Language Models over Instruction Tuningcited by 80
  6. 2024INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledgecited by 78
  7. 2023SILO Language Models: Isolating Legal Risk In a Nonparametric Datastorecited by 71
  8. 2023Scaling Expert Language Models with Unsupervised Domain Discoverycited by 59
  9. 2022Language Modelling with Pixelscited by 53
  10. 2023Adapters: A Unified Library for Parameter-Efficient and Modular Transfer Learningcited by 53
  11. 2023AutoPEFT: Automatic Configuration Search for Parameter-Efficient Fine-Tuningcited by 49
  12. 2023Knowledge Card: Filling LLMs' Knowledge Gaps with Plug-in Specialized Language Modelscited by 46
  13. 2023Improving Language Plasticity via Pretraining with Active Forgettingcited by 43
  14. 2024Segment Any Text: A Universal Approach for Robust, Efficient and Adaptable Sentence Segmentationcited by 41
  15. 2022NusaCrowd: Open Source Initiative for Indonesian NLP Resourcescited by 38
  16. 2023Emergent Modularity in Pre-trained Transformerscited by 37
  17. 2024Breaking the Curse of Multilinguality with Cross-lingual Expert Language Modelscited by 36
  18. 2024Gradient Routing: Masking Gradients to Localize Computation in Neural Networkscited by 28
  19. 2023Language Versatilists vs. Specialists: An Empirical Revisiting on Multilingual Transfer Abilitycited by 27
  20. 2024Model Swarms: Collaborative Search to Adapt LLM Experts via Swarm Intelligencecited by 27
  21. 2023CombLM: Adapting Black-Box Language Models through Small Fine-Tuned Modelscited by 26
  22. 2025Multilingual Routing in Mixture-of-Expertscited by 25
  23. 2022Phylogeny-Inspired Adaptation of Multilingual Models to New Languagescited by 24
  24. 2022Mini-Model Adaptation: Efficiently Extending Pretrained Models to New Languages via Aligned Shallow Trainingcited by 24
  25. 2023mmT5: Modular Multilingual Pre-Training Solves Source Language Hallucinationscited by 24
  26. 2024BertaQA: How Much Do Language Models Know About Local Culture?cited by 23
  27. 2023Romanization-based Large-scale Adaptation of Multilingual Language Modelscited by 21
  28. 2024IRCoder: Intermediate Representations Make Language Models Robust Multilingual Code Generatorscited by 21
  29. 2024Multi-Task Contrastive Learning for 8192-Token Bilingual Text Embeddingscited by 20
  30. 2024Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language Modelscited by 20
  31. 2023An Efficient Multilingual Language Model Compression through Vocabulary Trimmingcited by 19
  32. 2024Neuron Specialization: Leveraging intrinsic task modularity for multilingual machine translationcited by 19
  33. 2024Poro 34B and the Blessing of Multilingualitycited by 18
  34. 2023Where's the Point? Self-Supervised Multilingual Punctuation-Agnostic Sentence Segmentationcited by 17
  35. 2024On the Limitations of Compute Thresholds as a Governance Strategycited by 17
  36. 2024Efficiently Democratizing Medical LLMs for 50 Languages via a Mixture of Language Family Expertscited by 17
  37. 2023Transfer to a Low-Resource Language via Close Relatives: The Case Study on Faroesecited by 15
  38. 2023Multilingual DistilWhisper: Efficient Distillation of Multi-task Speech Models via Language-Specific Expertscited by 15

50 most cited of 145 All 145 (JSON)