mlsanityData to 7 Oct 2026

Continual Pre-training of Language Models

arXiv 2302.03241 ↗First posted 7 Feb 2023cs.CL cs.AI cs.LGcited by 194

Not on the map. Why?

Abstract (arXiv)

Language models (LMs) have been instrumental for the rapid advance of natural language processing. This paper studies continual pre-training of LMs, in particular, continual domain-adaptive pre-training (or continual DAP-training). Existing research has shown that further pre-training an LM using a domain corpus to adapt the LM to the domain can improve the end-task performance in the domain. This paper proposes a novel method to continually DAP-train an LM with a sequence of unlabeled domain corpora to adapt the LM to these domains to improve their end-task performances. The key novelty of our method is a soft-masking mechanism that directly controls the update to the LM. A novel proxy is also proposed to preserve the general knowledge in the original LM. Additionally, it contrasts the representations of the previously learned domain knowledge (including the general knowledge in the pre-trained LM) and the knowledge from the current full network to achieve knowledge integration. The method not only overcomes catastrophic forgetting, but also achieves knowledge transfer to improve end-task performances. Empirical evaluation demonstrates the effectiveness of the proposed method.

Key facts. Cited by 194 papers in the map (data through 7 Oct 2026). Builds on 54 of its 81 references that are in the map. First posted on arXiv 7 Feb 2023.

Builds on

54 of 81 references.

References · 54

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  6. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  9. 2016Pointer Sentinel Mixture Modelscited by 2,992
  10. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  11. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  12. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
38 more
  1. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  2. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  3. 2017Continual Learning with Deep Generative Replaycited by 1,152
  4. 2019Publicly Available Clinical BERT Embeddingscited by 995
  5. 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 895
  6. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  7. 2019Are Sixteen Heads Really Better than One?cited by 803
  8. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  9. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 672
  10. 2021Fast Model Editing at Scalecited by 491
  11. 2019How to Fine-Tune BERT for Text Classification?cited by 409
  12. 2018Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Constructioncited by 321
  13. 2020K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapterscited by 318
  14. 2016Dynamic Network Surgery for Efficient DNNscited by 233
  15. 2021PPT: Pre-trained Prompt Tuning for Few-shot Learningcited by 232
  16. 2020Supermasks in Superpositioncited by 223
  17. 2020Mnemonics Training: Multi-Class Incremental Learning without Forgettingcited by 220
  18. 2019BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysiscited by 176
  19. 2021Towards Continual Knowledge Learning of Language Modelscited by 172
  20. 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 130
  21. 2019A continual learning survey: Defying forgetting in classification taskscited by 118
  22. 2021Lifelong Pretraining: Continually Adapting Language Models to Emerging Corporacited by 114
  23. 2021Achieving Forgetting Prevention and Knowledge Transfer in Continual Learningcited by 91
  24. 2021Continual Learning for Text Classification with Information Disentanglement Based Regularizationcited by 86
  25. 2020Continual Learning in Task-Oriented Dialogue Systemscited by 82
  26. 2022A Theoretical Study on Solving Continual Learningcited by 78
  27. 2022ELLE: Efficient Lifelong Pre-training for Emerging Datacited by 70
  28. 2017Continual Learning in Generative Adversarial Netscited by 51
  29. 2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 50
  30. 2020Efficient Meta Lifelong-Learning with Limited Memorycited by 48
  31. 2021Differentiable Subset Pruning of Transformer Headscited by 46
  32. 2021Adapting BERT for Continual Learning of a Sequence of Aspect Sentiment Classification Taskscited by 46
  33. 2019Psycholinguistics meets Continual Learning: Measuring Catastrophic Forgetting in Visual Question Answeringcited by 33
  34. 2022Continual Training of Language Models for Few-Shot Learningcited by 33
  35. 2020Lifelong Language Knowledge Distillationcited by 31
  36. 2021CLASSIC: Continual and Contrastive Learning of Aspect Sentiment Classification Taskscited by 29
  37. 2019IMHO Fine-Tuning Improves Claim Detectioncited by 27
  38. 2021Should We Be Pre-training? An Argument for End-task Aware Training as an Alternativecited by 26

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 19; 2024: 75; 2025: 64; 2026: 36.

194 papers.

Sort:

By area:

  1. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Modelscited by 268
  2. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  3. 2023JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Modelscited by 170
  4. 2024Continual Learning for Large Language Models: A Surveycited by 144
  5. 2023Continual Pre-Training of Large Language Models: How to (re)warm your model?cited by 135
  6. 2024Simple and Scalable Strategies to Continually Pre-train Large Language Modelscited by 122
  7. 2023Hierarchical Decomposition of Prompt-Based Continual Learning: Rethinking Obscured Sub-optimalitycited by 121
  8. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 116
  9. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  10. 2024From Matching to Generation: A Survey on Generative Information Retrievalcited by 89
  11. 2024A Survey of Large Language Models in Finance (FinLLMs)cited by 77
  12. 2024RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Horizon Generationcited by 75
38 more
  1. 2024Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Modelscited by 69
  2. 2024TableGPT2: A Large Multimodal Model with Tabular Data Integrationcited by 69
  3. 2024Towards Lifelong Learning of Large Language Models: A Surveycited by 60
  4. 2024Investigating Continual Pretraining in Large Language Models: Insights and Implicationscited by 54
  5. 2025End-to-End Test-Time Training for Long Contextcited by 53
  6. 2023Knowledge Card: Filling LLMs' Knowledge Gaps with Plug-in Specialized Language Modelscited by 46
  7. 2024Recent Advances of Foundation Language Models-based Continual Learning: A Surveycited by 46
  8. 2024From Tokens to Words: On the Inner Lexicon of LLMscited by 46
  9. 2024D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Modelscited by 39
  10. 2025Scaling Agents via Continual Pre-trainingcited by 39
  11. 2024TelecomGPT: A Framework to Build Telecom-Specfic Large Language Modelscited by 38
  12. 2023Urban Generative Intelligence (UGI): A Foundational Platform for Agents in Embodied City Environmentcited by 37
  13. 2024AKEW: Assessing Knowledge Editing in the Wildcited by 35
  14. 2025Beyond Distillation: Pushing the Limits of Medical LLM Reasoning with Minimalist Rule-Based RLcited by 34
  15. 2023How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advancescited by 33
  16. 2023Efficient Continual Pre-training for Building Domain Specific Large Language Modelscited by 33
  17. 2024Selecting Large Language Model to Fine-tune via Rectified Scaling Lawcited by 33
  18. 2024SciLitLLM: How to Adapt LLMs for Scientific Literature Understandingcited by 29
  19. 2024MatryoshkaKV: Adaptive KV Compression via Trainable Orthogonal Projectioncited by 29
  20. 2024Code-Switching Curriculum Learning for Multilingual Transfer in LLMscited by 26
  21. 2024Set the Clock: Temporal Alignment of Pretrained Language Modelscited by 24
  22. 2025Demystifying Domain-adaptive Post-training for Financial LLMscited by 24
  23. 2024Unlocking Continual Learning Abilities in Language Modelscited by 22
  24. 2024Continual LLaVA: Continual Instruction Tuning in Large Vision-Language Modelscited by 22
  25. 2024More Than Catastrophic Forgetting: Integrating General Capabilities For Domain-Specific LLMscited by 21
  26. 2024Self-Updatable Large Language Models by Integrating Context into Model Parameterscited by 21
  27. 2025MoCa: Modality-aware Continual Pre-training Makes Better Bidirectional Multimodal Embeddingscited by 20
  28. 2024Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oraclecited by 19
  29. 2025A Survey on Agentic Multimodal Large Language Modelscited by 19
  30. 2023Class Incremental Learning via Likelihood Ratio Based Task Predictioncited by 17
  31. 2023ConPET: Continual Parameter-Efficient Tuning for Large Language Modelscited by 16
  32. 2024Examining Forgetting in Continual Pre-training of Aligned Large Language Modelscited by 16
  33. 2024A Survey on Large Language Models from General Purpose to Medical Applications: Datasets, Methodologies, and Evaluationscited by 16
  34. 2025Unlocking the Power of Function Vectors for Characterizing and Mitigating Catastrophic Forgetting in Continual Instruction Tuningcited by 16
  35. 2024Towards Effective and Efficient Continual Pre-training of Large Language Modelscited by 13
  36. 2024MoFO: Momentum-Filtered Optimizer for Mitigating Forgetting in LLM Fine-Tuningcited by 13
  37. 2024Towards LifeSpan Cognitive Systemscited by 13
  38. 2025Bisecle: Binding and Separation in Continual Learning for Video Language Understandingcited by 13

50 most cited of 194 All 194 (JSON)