mlsanityData to 14 Oct 2026

Soft Layer Selection with Meta-Learning for Zero-Shot Cross-Lingual Transfer

arXiv 2107.09840 ↗First posted 21 Jul 2021cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

Multilingual pre-trained contextual embedding models (Devlin et al., 2019) have achieved impressive performance on zero-shot cross-lingual transfer tasks. Finding the most effective fine-tuning strategy to fine-tune these models on high-resource languages so that it transfers well to the zero-shot languages is a non-trivial task. In this paper, we propose a novel meta-optimizer to soft-select which layers of the pre-trained model to freeze during fine-tuning. We train the meta-optimizer by simulating the zero-shot transfer scenario. Results on cross-lingual natural language inference show that our approach improves over the simple fine-tuning baseline and X-MAML (Nooralahzadeh et al., 2020).

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 31 of its 45 references that are in the map. First posted on arXiv 21 Jul 2021.

Builds on

31 of 45 references.

References · 31

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  4. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  5. 2016Matching Networks for One Shot Learningcited by 2,255
  6. 2016Neural Architecture Search with Reinforcement Learningcited by 1,499
  7. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,101
  8. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  9. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  10. 2016Learning to learn by gradient descent by gradient descentcited by 620
  11. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 360
  12. 2017Large-Scale Evolution of Image Classifierscited by 348
19 more
  1. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
  2. 2016Designing Neural Network Architectures using Reinforcement Learningcited by 279
  3. 2016Learning to Optimizecited by 195
  4. 2019Choosing Transfer Languages for Cross-Lingual Learningcited by 144
  5. 2018Cross-Lingual Transfer Learning for Multilingual Task Oriented Dialogcited by 125
  6. 2017Learned Optimizers that Scale and Generalizecited by 110
  7. 2018Meta-Learning for Low-Resource Neural Machine Translationcited by 105
  8. 2019Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsingcited by 70
  9. 2020Zero-Shot Cross-Lingual Transfer with Meta Learningcited by 66
  10. 2019Learning to Few-Shot Learn Across Diverse Natural Language Classification Taskscited by 57
  11. 2016Adversarial Deep Averaging Networks for Cross-Lingual Sentiment Classificationcited by 48
  12. 2019Attention-Informed Mixed-Language Training for Zero-shot Cross-lingual Task-oriented Dialogue Systemscited by 42
  13. 2020Meta-Learning for Domain Generalization in Semantic Parsingcited by 41
  14. 2019GluonCV and GluonNLP: Deep Learning in Computer Vision and Natural Language Processingcited by 32
  15. 2017Transfer Learning for Sequence Tagging with Hierarchical Recurrent Networkscited by 30
  16. 2019Enhanced Meta-Learning for Cross-lingual Named Entity Recognition with Minimal Resourcescited by 25
  17. 2018Zero-Shot Cross-lingual Classification Using Multilingual Neural Machine Translationcited by 19
  18. 2019Polyglot Contextual Representations Improve Crosslingual Transfercited by 16
  19. 2020Exploring Fine-tuning Techniques for Pre-trained Cross-lingual Models via Continual Learningcited by 9

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 3; 2023: 1.

5 papers.

  1. 2022Meta Learning for Natural Language Processing: A Surveycited by 26
  2. 2023Identifying the Correlation Between Language Distance and Cross-Lingual Transfer in a Multilingual Representation Spacecited by 12
  3. 2022Meta-Learning a Cross-lingual Manifold for Semantic Parsingcited by 11
  4. 2022Zero-Shot Dependency Parsing with Worst-Case Aware Automated Curriculum Learningcited by 5
  5. 2021Rule-based Morphological Inflection Improves Neural Terminology Translationcited by 3