mlsanityData to 7 Oct 2026

Examining Modularity in Multilingual LMs via Language-Specialized Subnetworks

arXiv 2311.08273 ↗First posted 14 Nov 2023cs.CLcited by 13

Not on the map. Why?

Abstract (arXiv)

Recent work has proposed explicitly inducing language-wise modularity in multilingual LMs via sparse fine-tuning (SFT) on per-language subnetworks as a means of better guiding cross-lingual sharing. In this work, we investigate (1) the degree to which language-wise modularity naturally arises within models with no special modularity interventions, and (2) how cross-lingual sharing and interference differ between such models and those with explicit SFT-guided subnetwork modularity. To quantify language specialization and cross-lingual interaction, we use a Training Data Attribution method that estimates the degree to which a model's predictions are influenced by in-language or cross-language training examples. Our results show that language-specialized subnetworks do naturally arise, and that SFT, rather than always increasing modularity, can decrease language specialization of subnetworks in favor of more cross-lingual sharing.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 38 of its 48 references that are in the map. First posted on arXiv 14 Nov 2023.

Builds on

38 of 48 references.

References · 38

  1. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  2. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  3. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  5. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  6. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  7. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  8. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
  9. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  10. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  11. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  12. 2019Are Sixteen Heads Really Better than One?cited by 803
26 more
  1. 2019How multilingual is Multilingual BERT?cited by 642
  2. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 565
  3. 2017Learning multiple visual domains with residual adapterscited by 553
  4. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 395
  5. 2014Sketching as a Tool for Numerical Linear Algebracited by 285
  6. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
  7. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 262
  8. 2018Efficient parametrization of multi-domain deep neural networkscited by 168
  9. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 145
  10. 2020Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functionscited by 144
  11. 2020The Multilingual Amazon Reviews Corpuscited by 142
  12. 2020FastIF: Scalable Influence Functions for Efficient Model Interpretation and Debuggingcited by 120
  13. 2020When BERT Plays the Lottery, All Tickets Are Winningcited by 114
  14. 2020Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Maskscited by 97
  15. 2016Learning to Compose Neural Networks for Question Answeringcited by 85
  16. 2020Zero-Shot Cross-Lingual Transfer with Meta Learningcited by 66
  17. 2021Learning Language Specific Sub-network for Multilingual Machine Translationcited by 48
  18. 2023Break It Down: Evidence for Structural Compositionality in Neural Networkscited by 44
  19. 2023Emergent Modularity in Pre-trained Transformerscited by 37
  20. 2019Learning Sparse Sharing Architectures for Multiple Taskscited by 31
  21. 2022Discovering Language-neutral Sub-networks in Multilingual Language Modelscited by 28
  22. 2023How do languages influence each other? Studying cross-lingual data sharing during LM fine-tuningcited by 14
  23. 2022Language Adaptive Cross-lingual Speech Representation Learning with Sparse Sharing Sub-networkscited by 12
  24. 2020Pair the Dots: Jointly Examining Training History and Test Stimuli for Model Interpretabilitycited by 7
  25. 2021On the Prunability of Attention Heads in Multilingual BERTcited by 4
  26. 2022Probing Structured Pruning on Multilingual Pre-trained Models: Settings, Algorithms, and Efficiencycited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 4; 2026: 1.

13 papers.

Sort:

By area:

  1. 2024The Same But Different: Structural Similarities and Differences in Multilingual Language Modelingcited by 27
  2. 2025Multilingual Routing in Mixture-of-Expertscited by 25
  3. 2024Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language Modelscited by 20
  4. 2024Neuron Specialization: Leveraging intrinsic task modularity for multilingual machine translationcited by 19
  5. 2024The Echoes of Multilinguality: Tracing Cultural Value Shifts during LM Fine-tuningcited by 10
  6. 2024The Heuristic Core: Understanding Subnetwork Generalization in Pretrained Language Modelscited by 9
  7. 2025Joint Localization and Activation Editing for Low-Resource Fine-Tuningcited by 7
  8. 2024Local Contrastive Editing of Gender Stereotypescited by 3
  9. 2026Extracting Small Translation Specialists from LLMs by Aggressively Pruning Expertscited by 3
  10. 2024Linguistically Grounded Analysis of Language Models using Shapley Head Valuescited by 1
  11. 2024Exploring Cross-lingual Latent Transplantation: Mutual Opportunities and Open Challengescited by 1
  12. 2025Utilizing Multilingual Encoders to Improve Large Language Models for Low-Resource Languagescited by 0
1 more
  1. 2025Exploring Training Data Attribution under Limited Access Constraintscited by 0