mlsanityData to 7 Oct 2026

Continual Learning Under Language Shift

arXiv 2311.01200 ↗First posted 2 Nov 2023cs.CL cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

The recent increase in data and model scale for language model pre-training has led to huge training costs. In scenarios where new data become available over time, updating a model instead of fully retraining it would therefore provide significant gains. We study the pros and cons of updating a language model when new data comes from new languages -- the case of continual learning under language shift. Starting from a monolingual English language model, we incrementally add data from Danish, Icelandic, and Norwegian to investigate how forward and backward transfer effects depend on pre-training order and characteristics of languages, for three different model sizes. Our results show that, while forward transfer is largely positive and independent of language order, backward transfer can be positive or negative depending on the order and characteristics of new languages. We explore a number of potentially explanatory factors and find that a combination of language contamination and syntactic similarity best fits our results.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 46 of its 61 references that are in the map. First posted on arXiv 2 Nov 2023.

Builds on

46 of 61 references.

References · 46

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  5. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  6. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  7. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  8. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  9. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  10. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  11. 2016Bag of Tricks for Efficient Text Classificationcited by 869
  12. 2019Online Continual Learning with Maximally Interfered Retrievalcited by 376
34 more
  1. 2021DyTox: Transformers for Continual Learning with DYnamic TOken eXpansioncited by 291
  2. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  3. 2021Mind the Gap: Assessing Temporal Generalization in Neural Language Modelscited by 226
  4. 2019REMIND Your Neural Network to Prevent Catastrophic Forgettingcited by 197
  5. 2023Continual Pre-training of Language Modelscited by 194
  6. 2019Episodic Memory in Lifelong Language Learningcited by 188
  7. 2021Towards Continual Knowledge Learning of Language Modelscited by 172
  8. 2020Continual Lifelong Learning in Natural Language Processing: A Surveycited by 152
  9. 2022Fine-tuned Language Models are Continual Learnerscited by 152
  10. 2020A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languagescited by 131
  11. 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 130
  12. 2022TimeLMs: Diachronic Language Models from Twittercited by 129
  13. 2019A continual learning survey: Defying forgetting in classification taskscited by 118
  14. 2022TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Modelscited by 117
  15. 2021Lifelong Pretraining: Continually Adapting Language Models to Emerging Corporacited by 114
  16. 2021Exploring the Limits of Large Scale Pre-trainingcited by 104
  17. 2022Language Contamination Helps Explain the Cross-lingual Capabilities of English Pretrained Modelscited by 90
  18. 2022Continual Learning of Natural Language Processing Tasks: A Surveycited by 83
  19. 2021What's in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpuscited by 79
  20. 2022ELLE: Efficient Lifelong Pre-training for Emerging Datacited by 70
  21. 2021Wide Neural Networks Forget Less Catastrophicallycited by 64
  22. 2021Continual Learning via Local Module Compositioncited by 61
  23. 2023Towards a Common Understanding of Contributing Factors for Cross-Lingual Transfer in Multilingual Language Models: A Reviewcited by 50
  24. 2020ECONET: Effective Continual Pretraining of Language Models for Event Temporal Reasoningcited by 38
  25. 2021Understanding Continual Learning Settings with Data Distribution Drift Analysiscited by 38
  26. 2021Towards Continual Learning for Multilingual Machine Translation via Vocabulary Substitutioncited by 31
  27. 2023Overcoming Catastrophic Forgetting in Massively Multilingual Continual Learningcited by 27
  28. 2021On Anytime Learning at Macroscalecited by 20
  29. 2022Challenging Common Assumptions about Catastrophic Forgettingcited by 20
  30. 2020The Danish Gigaword Projectcited by 17
  31. 2022Parameter-Efficient Finetuning for Robust Continual Multilingual Learningcited by 9
  32. 2023GPT-SW3: An Autoregressive Language Model for the Nordic Languagescited by 8
  33. 2023Training and Evaluation of a Multilingual Tokenizer for GPT-SW3cited by 7
  34. 2023The Nordic Pile: A 1.2TB Nordic Dataset for Language Modelingcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 4; 2026: 1.

9 papers.

  1. 2024Continual Learning for Large Language Models: A Surveycited by 144
  2. 2024Investigating Continual Pretraining in Large Language Models: Insights and Implicationscited by 54
  3. 2024Predicting positive transfer for improved low-resource speech recognition using acoustic pseudo-tokenscited by 11
  4. 2025TiEBe: Tracking Language Model Recall of Notable Worldwide Events Through Timecited by 8
  5. 2024BgGPT 1.0: Extending English-centric LLMs to other languagescited by 7
  6. 2025Building High-Quality Datasets for Portuguese LLMs: From Common Crawl Snapshots to Industrial-Grade Corporacited by 7
  7. 2025CultureMERT: Continual Pre-Training for Cross-Cultural Music Representation Learningcited by 1
  8. 2025XDoGE: Multilingual Data Reweighting to Enhance Language Inclusivity in LLMscited by 1
  9. 2026The Blessing of Dimensionality: How Near-Orthogonality in High-Dimensional Spaces Explains Temporal Portabilitycited by 0