mlsanityData to 7 Oct 2026

Data-Efficient Cross-Lingual Transfer with Language-Specific Subnetworks

arXiv 2211.00106 ↗First posted 31 Oct 2022cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

Large multilingual language models typically share their parameters across all languages, which enables cross-lingual task transfer, but learning can also be hindered when training updates from different languages are in conflict. In this paper, we propose novel methods for using language-specific subnetworks, which control cross-lingual parameter sharing, to reduce conflicts and increase positive transfer during fine-tuning. We introduce dynamic subnetworks, which are jointly updated with the model, and we combine our methods with meta-learning, an established, but complementary, technique for improving cross-lingual transfer. Finally, we provide extensive analyses of how each of our methods affects the models.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 40 of its 49 references that are in the map. First posted on arXiv 31 Oct 2022.

Builds on

40 of 49 references.

References · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  7. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  8. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  9. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  10. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  11. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
  12. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
28 more
  1. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  2. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  3. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  4. 2019Are Sixteen Heads Really Better than One?cited by 803
  5. 2019How multilingual is Multilingual BERT?cited by 642
  6. 2016Deep Biaffine Attention for Neural Dependency Parsingcited by 334
  7. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
  8. 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 253
  9. 2020When BERT Plays the Lottery, All Tickets Are Winningcited by 114
  10. 2020Finding Universal Grammatical Relations in Multilingual BERTcited by 108
  11. 2018Meta-Learning for Low-Resource Neural Machine Translationcited by 105
  12. 201975 Languages, 1 Model: Parsing Universal Dependencies Universallycited by 90
  13. 2019Do Attention Heads in BERT Track Syntactic Dependencies?cited by 87
  14. 2020UDapter: Language Adaptation for Truly Universal Dependency Parsingcited by 77
  15. 2020Zero-Shot Cross-Lingual Transfer with Meta Learningcited by 66
  16. 2019The Generalization-Stability Tradeoff In Neural Network Pruningcited by 54
  17. 2021Lightweight Adapter Tuning for Multilingual Speech Translationcited by 53
  18. 2022Analyzing the Mono- and Cross-Lingual Pretraining Dynamics of Multilingual Language Modelscited by 32
  19. 2019Learning Sparse Sharing Architectures for Multiple Taskscited by 31
  20. 2022Discovering Language-neutral Sub-networks in Multilingual Language Modelscited by 28
  21. 2019Enhanced Meta-Learning for Cross-lingual Named Entity Recognition with Minimal Resourcescited by 25
  22. 2020On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatmentcited by 21
  23. 2019Zero-shot Dependency Parsing with Pre-trained Multilingual Sentence Representationscited by 17
  24. 2021Sequential Reptile: Inter-Task Gradient Alignment for Multilingual Learningcited by 16
  25. 2022Language Adaptive Cross-lingual Speech Representation Learning with Sparse Sharing Sub-networkscited by 12
  26. 2022Learning ASR pathways: A sparse multilingual ASR modelcited by 12
  27. 2021Meta-Learning for Fast Cross-Lingual Adaptation in Dependency Parsingcited by 10
  28. 2022Improving the Cross-Lingual Generalisation in Visual Question Answeringcited by 5

Built on by

Cited by year

Citing papers per year: 2023: 2.

2 papers.

  1. 2023Modular Deep Learningcited by 82
  2. 2023How do languages influence each other? Studying cross-lingual data sharing during LM fine-tuningcited by 14