mlsanityData to 14 Oct 2026

Discovering Language-neutral Sub-networks in Multilingual Language Models

arXiv 2205.12672 ↗First posted 25 May 2022cs.CLcited by 28

Not on the map. Why?

Abstract (arXiv)

Multilingual pre-trained language models transfer remarkably well on cross-lingual downstream tasks. However, the extent to which they learn language-neutral representations (i.e., shared representations that encode similar phenomena across languages), and the effect of such representations on cross-lingual transfer performance, remain open questions. In this work, we conceptualize language neutrality of multilingual models as a function of the overlap between language-encoding sub-networks of these models. We employ the lottery ticket hypothesis to discover sub-networks that are individually optimized for various languages and tasks. Our evaluation across three distinct tasks and eleven typologically-diverse languages demonstrates that sub-networks for different languages are topologically similar (i.e., language-neutral), making them effective initializations for cross-lingual transfer with limited performance degradation.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Builds on 41 of its 49 references that are in the map. First posted on arXiv 25 May 2022.

Builds on

41 of 49 references.

References · 41

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  6. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  7. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  8. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  9. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  10. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  11. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  12. 2019Cross-lingual Language Model Pretrainingcited by 1,090
29 more
  1. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  2. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  3. 2019How multilingual is Multilingual BERT?cited by 645
  4. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 637
  5. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
  6. 2019The State of Sparsity in Deep Neural Networkscited by 476
  7. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 340
  8. 2018Insights on representational similarity in neural networks with canonical correlationcited by 324
  9. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
  10. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
  11. 2021Training Neural Networks with Fixed Sparse Maskscited by 214
  12. 2019Cross-Lingual Ability of Multilingual BERT: An Empirical Studycited by 182
  13. 2019Emerging Cross-lingual Structure in Pretrained Language Modelscited by 181
  14. 2021Composable Sparse Fine-Tuning for Cross-Lingual Transfercited by 144
  15. 2020When BERT Plays the Lottery, All Tickets Are Winningcited by 114
  16. 2020What Happens To BERT Embeddings During Fine-tuning?cited by 113
  17. 2020Finding Universal Grammatical Relations in Multilingual BERTcited by 110
  18. 2019One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizerscited by 102
  19. 2020On the Language Neutrality of Pre-trained Multilingual Representationscited by 85
  20. 2018Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddingscited by 83
  21. 2019Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLPcited by 73
  22. 2021First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERTcited by 65
  23. 2019How Language-Neutral is Multilingual BERT?cited by 51
  24. 2020Inducing Language-Agnostic Multilingual Representationscited by 48
  25. 2021Learning Language Specific Sub-network for Multilingual Machine Translationcited by 48
  26. 2020What's so special about BERT's layers? A closer look at the NLP pipeline in monolingual and multilingual modelscited by 35
  27. 2020It's not Greek to mBERT: Inducing Word-Level Translations from Multilingual BERTcited by 30
  28. 2020What does it mean to be language-agnostic? Probing multilingual sentence encoders for typological propertiescited by 23
  29. 2020On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatmentcited by 21

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 12; 2024: 8; 2025: 1; 2026: 5.

28 papers.

Sort:

By area:

  1. 2024Investigating Cultural Alignment of Large Language Modelscited by 143
  2. 2023Modular Deep Learningcited by 83
  3. 2024Probing the Emergence of Cross-lingual Alignment during LLM Trainingcited by 39
  4. 2024Interpreting Arithmetic Mechanism in Large Language Models through Comparative Neuron Analysiscited by 31
  5. 2023Do Localization Methods Actually Localize Memorized Data in LLMs? A Tale of Two Benchmarkscited by 29
  6. 2024Goldfish: Monolingual Language Models for 350 Languagescited by 29
  7. 2024Opportunities and Challenges of Large Language Models for Low-Resource Languages in Humanities Researchcited by 27
  8. 2023Discovering Knowledge-Critical Subnetworks in Pretrained Language Modelscited by 16
  9. 2023Cross-Lingual Transfer with Target Language-Ready Task Adapterscited by 15
  10. 2023How do languages influence each other? Studying cross-lingual data sharing during LM fine-tuningcited by 14
  11. 2024Tracing the Roots of Facts in Multilingual Language Models: Independent, Shared, and Transferred Knowledgecited by 14
  12. 2023Examining Modularity in Multilingual LMs via Language-Specialized Subnetworkscited by 13
16 more
  1. 2024Analyzing the Evaluation of Cross-Lingual Knowledge Transfer in Multilingual Language Modelscited by 11
  2. 2025The Translation Barrier Hypothesis: Multilingual Generation with Large Language Models Suffers from Implicit Translation Failurecited by 10
  3. 2022Improving the Cross-Lingual Generalisation in Visual Question Answeringcited by 5
  4. 2023A Joint Matrix Factorization Analysis of Multilingual Representationscited by 5
  5. 2023Breaking the Language Barrier: Improving Cross-Lingual Reasoning with Structured Self-Attentioncited by 4
  6. 2023Dynamic ASR Pathways: An Adaptive Masking Approach Towards Efficient Pruning of A Multilingual ASR Modelcited by 3
  7. 2022Data-Efficient Cross-Lingual Transfer with Language-Specific Subnetworkscited by 2
  8. 2023Advancing Perception in Artificial Intelligence through Principles of Cognitive Sciencecited by 1
  9. 2024Linguistically Grounded Analysis of Language Models using Shapley Head Valuescited by 1
  10. 2023Multilingual Text Representationcited by 0
  11. 2023X-SNS: Cross-Lingual Transfer Prediction through Sub-Network Similaritycited by 0
  12. 2026From National Curricula to Cultural Awareness: Constructing Open-Ended Culture-Specific Question Answering Datasetcited by 0
  13. 2026Universal Conceptual Structure in Neural Translation: Probing NLLB-200's Multilingual Geometrycited by 0
  14. 2026Computational Lesions in Multilingual Language Models Separate Shared and Language-specific Brain Alignmentcited by 0
  15. 2026Lost in the Tower of Babel: The Adverse Effects of Incidental Multilingualism in LLMscited by 0
  16. 2026Knowledge Offloading: Decomposing LLMs into Sparse Backbones and Memory Modulescited by 0