mlsanityData to 7 Oct 2026

Continual Pre-Training Mitigates Forgetting in Language and Vision

arXiv 2205.09357 ↗First posted 19 May 2022cs.LG cs.AIcited by 82

Not on the map. Why?

Abstract (arXiv)

Pre-trained models are nowadays a fundamental component of machine learning research. In continual learning, they are commonly used to initialize the model before training on the stream of non-stationary data. However, pre-training is rarely applied during continual learning. We formalize and investigate the characteristics of the continual pre-training scenario in both language and vision environments, where a model is continually pre-trained on a stream of incoming data and only later fine-tuned to different downstream tasks. We show that continually pre-trained models are robust against catastrophic forgetting and we provide strong empirical evidence supporting the fact that self-supervised pre-training is more effective in retaining previous knowledge than supervised protocols. Code is provided at https://github.com/AndreaCossu/continual-pretraining-nlp-vision .

Key facts. Cited by 82 papers in the map (data through 7 Oct 2026). Builds on 39 of its 54 references that are in the map. First posted on arXiv 19 May 2022.

Builds on

39 of 54 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  9. 2016Pointer Sentinel Mixture Modelscited by 2,992
  10. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  11. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  12. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
27 more
  1. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  2. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  3. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  4. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  5. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  6. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
  7. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  8. 2021DyTox: Transformers for Continual Learning with DYnamic TOken eXpansioncited by 291
  9. 2017CORe50: a New Dataset and Benchmark for Continuous Object Recognitioncited by 287
  10. 2020Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depthcited by 241
  11. 2021Mind the Gap: Assessing Temporal Generalization in Neural Language Modelscited by 226
  12. 2021Towards Continual Knowledge Learning of Language Modelscited by 172
  13. 2021Self-Supervised Models are Continual Learnerscited by 149
  14. 2021An Empirical Investigation of the Role of Pre-training in Lifelong Learningcited by 134
  15. 2022TimeLMs: Diachronic Language Models from Twittercited by 129
  16. 2021Avalanche: an End-to-End Library for Continual Learningcited by 124
  17. 2022TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Modelscited by 117
  18. 2021Lifelong Pretraining: Continually Adapting Language Models to Emerging Corporacited by 114
  19. 2021Representational Continuity for Unsupervised Continual Learningcited by 95
  20. 2019Continual Learning for Robotics: Definition, Framework, Learning Strategies, Opportunities and Challengescited by 92
  21. 2022Probing Representation Forgetting in Supervised and Unsupervised Continual Learningcited by 74
  22. 2022ELLE: Efficient Lifelong Pre-training for Emerging Datacited by 70
  23. 2020ECONET: Effective Continual Pretraining of Language Models for Event Temporal Reasoningcited by 38
  24. 2021How Well Does Self-Supervised Pre-Training Perform with Streaming Data?cited by 29
  25. 2020Multi-Stage Pre-training for Low-Resource Domain Adaptationcited by 24
  26. 2022Practical Recommendations for Replay-based Continual Learning Methodscited by 16
  27. 2020Continual Learning using a Bayesian Nonparametric Dictionary of Weight Factorscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 13; 2024: 35; 2025: 22; 2026: 9.

82 papers.

Sort:

By area:

  1. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  2. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  3. 2023SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Modelcited by 178
  4. 2025RL's Razor: Why Online Reinforcement Learning Forgets Lesscited by 162
  5. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 158
  6. 2024Eight Methods to Evaluate Robust Unlearning in LLMscited by 153
  7. 2022Fine-tuned Language Models are Continual Learnerscited by 152
  8. 2024Continual Learning for Large Language Models: A Surveycited by 144
  9. 2023Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Maskscited by 130
  10. 2024Simple and Scalable Strategies to Continually Pre-train Large Language Modelscited by 122
  11. 2022SuS-X: Training-Free Name-Only Transfer of Vision-Language Modelscited by 118
  12. 2024From Matching to Generation: A Survey on Generative Information Retrievalcited by 89
38 more
  1. 2024Defending Against Unforeseen Failure Modes with Latent Adversarial Trainingcited by 82
  2. 2024Towards Lifelong Learning of Large Language Models: A Surveycited by 60
  3. 2023Continual Learning: Applications and the Road Forwardcited by 59
  4. 2024Augmenting Math Word Problems via Iterative Question Composingcited by 58
  5. 2024Investigating Continual Pretraining in Large Language Models: Insights and Implicationscited by 54
  6. 2024Recent Advances of Foundation Language Models-based Continual Learning: A Surveycited by 46
  7. 2024D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Modelscited by 39
  8. 2024Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Modelscited by 29
  9. 2023A Comprehensive Empirical Evaluation on Online Continual Learningcited by 24
  10. 2024CMR Scaling Law: Predicting Critical Mixture Ratios for Continual Pre-training of Language Modelscited by 23
  11. 2024Continual LLaVA: Continual Instruction Tuning in Large Vision-Language Modelscited by 22
  12. 2024More Than Catastrophic Forgetting: Integrating General Capabilities For Domain-Specific LLMscited by 21
  13. 2024Self-Updatable Large Language Models by Integrating Context into Model Parameterscited by 21
  14. 2025MLLM-CL: Continual Learning for Multimodal Large Language Modelscited by 21
  15. 2024Salute the Classic: Revisiting Challenges of Machine Translation in the Age of Large Language Modelscited by 17
  16. 2023Knowledge Accumulation in Continually Learned Representations and the Issue of Feature Forgettingcited by 16
  17. 2025The Future of Continual Learning in the Era of Foundation Models: Three Key Directionscited by 16
  18. 2023Class Incremental Learning with Pre-trained Vision-Language Modelscited by 15
  19. 2024Towards LifeSpan Cognitive Systemscited by 13
  20. 2025Bisecle: Binding and Separation in Continual Learning for Video Language Understandingcited by 13
  21. 2024Gold Panning in Vocabulary: An Adaptive Method for Vocabulary Expansion of Domain-Specific LLMscited by 11
  22. 2024Delving into the Reversal Curse: How Far Can Large Language Models Generalize?cited by 11
  23. 2023Continual Learning for Predictive Maintenance: Overview and Challengescited by 10
  24. 2024SLCA++: Unleash the Power of Sequential Fine-tuning for Continual Learning with Pre-trainingcited by 10
  25. 2025Krutrim LLM: Multilingual Foundational Model for over a Billion Peoplecited by 9
  26. 2025Bring Your Own Knowledge: A Survey of Methods for LLM Knowledge Expansioncited by 8
  27. 2025SinLlama -- A Large Language Model for Sinhalacited by 8
  28. 2026Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averagingcited by 8
  29. 2024Continual Learning with Neuromorphic Computing: Foundations, Methods, and Emerging Applicationscited by 7
  30. 2024An Empirical Analysis of Forgetting in Pre-trained Models with Incremental Low-Rank Updatescited by 6
  31. 2024How green is continual learning, really? Analyzing the energy consumption in continual training of vision foundation modelscited by 5
  32. 2025Continual Pre-training of MoEs: How robust is your router?cited by 5
  33. 2022Addressing Distribution Shift at Test Time in Pre-trained Language Modelscited by 4
  34. 2025Enhancing Domain-Specific Encoder Models with LLM-Generated Data: How to Leverage Ontologies, and How to Do Without Themcited by 4
  35. 2024Gradient Localization Improves Lifelong Pretraining of Language Modelscited by 3
  36. 2024ModelGrow: Continual Text-to-Video Pre-training with Model Expansion and Language Understanding Enhancementcited by 3
  37. 2025MoL for LLMs: Dual-Loss Optimization to Enhance Domain Expertise While Preserving General Capabilitiescited by 3
  38. 2026Small LLMs for Medical NLP: a Systematic Analysis of Few-Shot, Constraint Decoding, Fine-Tuning and Continual Pre-Training in Italiancited by 3

50 most cited of 82 All 82 (JSON)