mlsanityData to 14 Oct 2026

Cross-lingual Lifelong Learning

arXiv 2205.11152 ↗First posted 23 May 2022cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

The longstanding goal of multi-lingual learning has been to develop a universal cross-lingual model that can withstand the changes in multi-lingual data distributions. There has been a large amount of work to adapt such multi-lingual models to unseen target languages. However, the majority of work in this direction focuses on the standard one-hop transfer learning pipeline from source to target languages, whereas in realistic scenarios, new languages can be incorporated at any time in a sequential manner. In this paper, we present a principled Cross-lingual Continual Learning (CCL) evaluation paradigm, where we analyze different categories of approaches used to continually adapt to emerging data from different languages. We provide insights into what makes multilingual sequential learning particularly challenging. To surmount such challenges, we benchmark a representative set of cross-lingual continual learning algorithms and analyze their knowledge preservation, accumulation, and generalization capabilities compared to baselines on carefully curated datastreams. The implications of this analysis include a recipe for how to measure and balance different cross-lingual continual learning desiderata, which go beyond conventional transfer learning.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 34 of its 59 references that are in the map. First posted on arXiv 23 May 2022.

Builds on

34 of 59 references.

References · 34

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  6. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  7. 2016Learning without Forgettingcited by 2,496
  8. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  9. 2017Continual Learning with Deep Generative Replaycited by 1,158
  10. 2018Efficient Lifelong Learning with A-GEMcited by 1,058
  11. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  12. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
22 more
  1. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
  2. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 445
  3. 2020AdapterHub: A Framework for Adapting Transformerscited by 444
  4. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 340
  5. 2019On Tiny Episodic Memories in Continual Learningcited by 307
  6. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 264
  7. 2018Online Structured Laplace Approximations For Overcoming Catastrophic Forgettingcited by 201
  8. 2019Episodic Memory in Lifelong Language Learningcited by 188
  9. 2019LAMOL: LAnguage MOdeling for Lifelong Language Learningcited by 160
  10. 2020MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmarkcited by 158
  11. 2020Continual Lifelong Learning in Natural Language Processing: A Surveycited by 152
  12. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 146
  13. 2020When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Modelscited by 131
  14. 2021Lifelong Pretraining: Continually Adapting Language Models to Emerging Corporacited by 115
  15. 2020UNKs Everywhere: Adapting Multilingual Language Models to New Scriptscited by 113
  16. 2021WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language modelscited by 89
  17. 2019Knowledge Distillation from Internal Representationscited by 87
  18. 2020Continual Learning in Task-Oriented Dialogue Systemscited by 83
  19. 2020More Data, More Relations, More Context and More Openness: A Review and Outlook for Relation Extractioncited by 52
  20. 2021Towards Continual Learning for Multilingual Machine Translation via Vocabulary Substitutioncited by 31
  21. 2019Multi-lingual Intent Detection and Slot Filling in a Joint BERT-based Modelcited by 24
  22. 2021X-METRA-ADA: Cross-lingual Meta-Transfer Learning Adaptation to Natural Language Understanding and Question Answeringcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 1; 2024: 2; 2025: 1; 2026: 3.

9 papers.

  1. 2023Overcoming Catastrophic Forgetting in Massively Multilingual Continual Learningcited by 27
  2. 2024Akal Badi ya Bias: An Exploratory Study of Gender Bias in Hindi Language Technologycited by 17
  3. 2022Parameter-Efficient Finetuning for Robust Continual Multilingual Learningcited by 9
  4. 2024Exploring Continual Learning of Compositional Generalization in NLIcited by 4
  5. 2025What Causes Knowledge Loss in Multilingual Language Models?cited by 2
  6. 2026Continual-learning for Modelling Low-Resource Languages from Large Language Modelscited by 1
  7. 2022On Robust Incremental Learning over Many Multilingual Stepscited by 0
  8. 2026Cross-Lingual Probing and Community-Grounded Analysis of Gender Bias in Low-Resource Bengalicited by 0
  9. 2026Structural Abstraction as an Inductive Bias for Non-Stationary Language Model Trainingcited by 0