Analyzing and Reducing the Performance Gap in Cross-Lingual Transfer with Fine-tuning Slow and Fast
Not on the map. Why?
Abstract (arXiv)
Existing research has shown that a multilingual pre-trained language model fine-tuned with one (source) language also performs well on downstream tasks for non-source languages, even though no fine-tuning is done on these languages. However, there is a clear gap between the performance of the source language and that of the non-source languages. This paper analyzes the fine-tuning process, discovers when the performance gap changes and identifies which network weights affect the overall performance most. Additionally, the paper seeks to answer to what extent the gap can be reduced by reducing forgetting. Based on the analysis results, a method named Fine-tuning slow and fast with four training policies is proposed to address these issues. Experimental results show the proposed method outperforms baselines by a clear margin.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 27 of its 38 references that are in the map. First posted on arXiv 19 May 2023.
Builds on
27 of 38 references.
References · 27
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
- 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
- 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
- 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
- 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
- 2019Similarity of Neural Network Representations Revisitedcited by 1,755
- 2018Experience Replay for Continual Learningcited by 1,095
- 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
- 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 911
- 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
- 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 340
15 more
- 2020Gradient Starvation: A Learning Proclivity in Neural Networkscited by 267
- 2020InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Trainingcited by 237
- 2020XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generationcited by 222
- 2020Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgettingcited by 170
- 2021On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptationcited by 147
- 2021Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuningcited by 146
- 2022Memory Replay with Data Compression for Continual Learningcited by 82
- 2020ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corporacited by 63
- 2020Explicit Alignment Objectives for Multilingual Bidirectional Encoderscited by 50
- 2022NoisyTune: A Little Noise Can Help You Finetune Pretrained Language Models Bettercited by 43
- 2021Consistency Regularization for Cross-Lingual Fine-Tuningcited by 41
- 2020FILTER: An Enhanced Fusion Method for Cross-lingual Language Understandingcited by 29
- 2020Orthogonal Language and Task Adapters in Zero-Shot Cross-Lingual Transfercited by 23
- 2021Sequential Reptile: Inter-Task Gradient Alignment for Multilingual Learningcited by 16
- 2022ROSE: Robust Selective Fine-tuning for Pre-trained Language Modelscited by 6
Built on by
Citing papers per year: 2024: 1; 2025: 1.
2 papers.