mlsanityData to 7 Oct 2026

Parameter-Efficient Finetuning for Robust Continual Multilingual Learning

arXiv 2209.06767 ↗First posted 14 Sep 2022cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

We introduce and study the problem of Continual Multilingual Learning (CML) where a previously trained multilingual model is periodically updated using new data arriving in stages. If the new data is present only in a subset of languages, we find that the resulting model shows improved performance only on the languages included in the latest update (and a few closely related languages) while its performance on all the remaining languages degrade significantly. We address this challenge by proposing LAFT-URIEL, a parameter-efficient finetuning strategy which aims to increase the number of languages on which the model improves after an update, while reducing the magnitude of loss in performance for the remaining languages. LAFT-URIEL uses linguistic knowledge to balance overfitting and knowledge sharing across languages, allowing for an additional 25% of task languages to see an improvement in performance after an update, while also reducing the average magnitude of losses on the remaining languages by 78% relative.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 24 of its 39 references that are in the map. First posted on arXiv 14 Sep 2022.

Builds on

24 of 39 references.

References · 24

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  6. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  7. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  8. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  9. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  10. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  11. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 443
  12. 2020AdapterHub: A Framework for Adapting Transformerscited by 441
12 more
  1. 2019Leveraging Pre-trained Checkpoints for Sequence Generation Taskscited by 182
  2. 2020Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgettingcited by 169
  3. 2020MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmarkcited by 156
  4. 2021MasakhaNER: Named Entity Recognition for African Languagescited by 155
  5. 2021Composable Sparse Fine-Tuning for Cross-Lingual Transfercited by 144
  6. 2021Adapting BERT for Continual Learning of a Sequence of Aspect Sentiment Classification Taskscited by 46
  7. 2021Towards Continual Learning for Multilingual Machine Translation via Vocabulary Substitutioncited by 31
  8. 2022Combining Modular Skills in Multitask Learningcited by 27
  9. 2021Continual Learning in Multilingual NMT via Language-Specific Embeddingscited by 12
  10. 2021DiS-ReX: A Multilingual Dataset for Distantly Supervised Relation Extractioncited by 9
  11. 2022Cross-lingual Lifelong Learningcited by 9
  12. 2021Towards Lifelong Learning of Multilingual Text-To-Speech Synthesiscited by 2

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 4; 2025: 3.

9 papers.

  1. 2024Towards Lifelong Learning of Large Language Models: A Surveycited by 60
  2. 2023Language Models are Universal Embedderscited by 11
  3. 2024RST-LoRA: A Discourse-Aware Low-Rank Adaptation for Long Document Abstractive Summarizationcited by 10
  4. 2025Continual Gradient Low-Rank Projection Fine-Tuning for LLMscited by 10
  5. 2023Continual Learning Under Language Shiftcited by 9
  6. 2024How Far Can In-Context Alignment Go? Exploring the State of In-Context Alignmentcited by 6
  7. 2025Emergent Abilities of Large Language Models under Continued Pretraining for Language Adaptationcited by 5
  8. 2025What Causes Knowledge Loss in Multilingual Language Models?cited by 2
  9. 2024Exploring Continual Fine-Tuning for Enhancing Language Ability in Large Language Modelcited by 0