mlsanityData to 14 Oct 2026

Overcoming Catastrophic Forgetting in Massively Multilingual Continual Learning

arXiv 2305.16252 ↗First posted 25 May 2023cs.CLcited by 27

Not on the map. Why?

Abstract (arXiv)

Real-life multilingual systems should be able to efficiently incorporate new languages as data distributions fed to the system evolve and shift over time. To do this, systems need to handle the issue of catastrophic forgetting, where the model performance drops for languages or tasks seen further in its past. In this paper, we study catastrophic forgetting, as well as methods to minimize this, in a massively multilingual continual learning framework involving up to 51 languages and covering both classification and sequence labeling tasks. We present LR ADJUST, a learning rate scheduling method that is simple, yet effective in preserving new information without strongly overwriting past knowledge. Furthermore, we show that this method is effective across multiple continual learning approaches. Finally, we provide further insights into the dynamics of catastrophic forgetting in this massively multilingual setup.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Builds on 15 of its 28 references that are in the map. First posted on arXiv 25 May 2023.

Builds on

15 of 28 references.

References · 15

  1. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  2. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  3. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  4. 2016Progressive Neural Networkscited by 1,361
  5. 2018Experience Replay for Continual Learningcited by 1,095
  6. 2018Efficient Lifelong Learning with A-GEMcited by 1,058
  7. 2019Episodic Memory in Lifelong Language Learningcited by 188
  8. 2020Continual Lifelong Learning in Natural Language Processing: A Surveycited by 152
  9. 2022MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languagescited by 146
  10. 2020Continual Learning in Task-Oriented Dialogue Systemscited by 83
  11. 2019The Step Decay Schedule: A Near Optimal, Geometrically Decaying Learning Rate Procedure For Least Squarescited by 68
  12. 2021Learn Continually, Generalize Rapidly: Lifelong Knowledge Accumulation for Few-shot Learningcited by 35
3 more
  1. 2020Exploring Fine-tuning Techniques for Pre-trained Cross-lingual Models via Continual Learningcited by 9
  2. 2022Cross-lingual Lifelong Learningcited by 9
  3. 2023Average Is Not Enough: Caveats of Multilingual Evaluationcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 13; 2025: 10; 2026: 2.

27 papers.

Sort:

By area:

  1. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 234
  2. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 186
  3. 2023Continual Pre-Training of Large Language Models: How to (re)warm your model?cited by 137
  4. 2024The Emerged Security and Privacy of LLM Agent: A Survey with Case Studiescited by 131
  5. 2025A Survey of Context Engineering for Large Language Modelscited by 126
  6. 2024Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Modelscited by 72
  7. 2024Towards Lifelong Learning of Large Language Models: A Surveycited by 64
  8. 2024Breaking the Curse of Multilinguality with Cross-lingual Expert Language Modelscited by 38
  9. 2024Opportunities and Challenges of Large Language Models for Low-Resource Languages in Humanities Researchcited by 27
  10. 2024Breaking Language Barriers: Cross-Lingual Continual Pre-Training at Scalecited by 15
  11. 2024PanGu-π Pro:Rethinking Optimization and Architecture for Tiny Language Modelscited by 14
  12. 2024SLCA++: Unleash the Power of Sequential Fine-tuning for Continual Learning with Pre-trainingcited by 10
15 more
  1. 2025Conditions for Catastrophic Forgetting in Multilingual Translationcited by 10
  2. 2023Continual Learning Under Language Shiftcited by 9
  3. 2024Towards Rehearsal-Free Multilingual ASR: A LoRA-based Case Study on Whispercited by 9
  4. 2024Demystifying Language Model Forgetting with Low-rank Example Associationscited by 8
  5. 2025Beyond Cosine Decay: On the effectiveness of Infinite Learning Rate Schedule for Continual Pre-trainingcited by 7
  6. 2025Continually Adding New Languages to Multilingual Language Modelscited by 7
  7. 2025Full-Parameter Continual Pretraining of Gemma2: Insights into Fluency and Domain Knowledgecited by 4
  8. 2025Mitigating Catastrophic Forgetting in Target Language Adaptation of LLMs via Source-Shielded Updatescited by 4
  9. 2025High-Dimensional Interlingual Representations of Large Language Modelscited by 2
  10. 2025What Causes Knowledge Loss in Multilingual Language Models?cited by 2
  11. 2025Improving Continual Pre-training Through Seamless Data Packingcited by 1
  12. 2024Exploring Continual Fine-Tuning for Enhancing Language Ability in Large Language Modelcited by 0
  13. 2025Sequencing to Mitigate Catastrophic Forgetting in Continual Learningcited by 0
  14. 2026Structural Abstraction as an Inductive Bias for Non-Stationary Language Model Trainingcited by 0
  15. 2026Towards Scalable Customization and Deployment of Multi-Agent Systems for Enterprise Applicationscited by 0