mlsanityData to 7 Oct 2026

Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora

arXiv 2110.08534 ↗First posted 16 Oct 2021cs.CLcited by 114

Not on the map. Why?

Abstract (arXiv)

Pretrained language models (PTLMs) are typically learned over a large, static corpus and further fine-tuned for various downstream tasks. However, when deployed in the real world, a PTLM-based model must deal with data distributions that deviate from what the PTLM was initially trained on. In this paper, we study a lifelong language model pretraining challenge where a PTLM is continually updated so as to adapt to emerging data. Over a domain-incremental research paper stream and a chronologically-ordered tweet stream, we incrementally pretrain a PTLM with different continual learning algorithms, and keep track of the downstream task performance (after fine-tuning). We evaluate PTLM's ability to adapt to new corpora while retaining learned knowledge in earlier corpora. Our experiments show distillation-based approaches to be most effective in retaining downstream performance in earlier domains. The algorithms also improve knowledge transfer, allowing models to achieve better downstream performance over the latest data, and improve temporal generalization when distribution gaps exist between training and evaluation because of time. We believe our problem formulation, methods, and analysis will inspire future studies towards continual pretraining of language models.

Key facts. Cited by 114 papers in the map (data through 7 Oct 2026). Builds on 39 of its 64 references that are in the map. First posted on arXiv 16 Oct 2021.

Builds on

39 of 64 references.

References · 39

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  7. 2016Learning without Forgettingcited by 2,465
  8. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  9. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  10. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  11. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  12. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
27 more
  1. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  2. 2018Progress & Compress: A scalable framework for continual learningcited by 516
  3. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  4. 2020BERTweet: A pre-trained language model for English Tweetscited by 346
  5. 2018Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Constructioncited by 321
  6. 2020K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapterscited by 318
  7. 2021Time-Aware Language Models as Temporal Knowledge Basescited by 290
  8. 2021Co2L: Contrastive Continual Learningcited by 243
  9. 2019Episodic Memory in Lifelong Language Learningcited by 188
  10. 2021Towards Continual Knowledge Learning of Language Modelscited by 172
  11. 2019LAMOL: LAnguage MOdeling for Lifelong Language Learningcited by 156
  12. 2021SEED: Self-supervised Distillation For Visual Representationcited by 132
  13. 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 130
  14. 2017PubMed 200k RCT: a Dataset for Sequential Sentence Classification in Medical Abstractscited by 123
  15. 2017SemEval 2017 Task 10: ScienceIE - Extracting Keyphrases and Relations from Scientific Publicationscited by 101
  16. 2020COVID-Twitter-BERT: A Natural Language Processing Model to Analyse COVID-19 Content on Twittercited by 96
  17. 2021Continual Learning for Text Classification with Information Disentanglement Based Regularizationcited by 86
  18. 2019Sentence Embedding Alignment for Lifelong Relation Extractioncited by 70
  19. 2021Temporal Adaptation of BERT and Performance on Downstream Document Classification: Insights from Social Mediacited by 51
  20. 2019The Materials Science Procedural Text Corpus: Annotating Materials Synthesis Procedures with Shallow Semantic Structurescited by 48
  21. 2020Efficient Meta Lifelong-Learning with Limited Memorycited by 48
  22. 2021Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domainscited by 36
  23. 2020Lifelong Language Knowledge Distillationcited by 31
  24. 2021Dynamic Language Models for Continuously Evolving Contentcited by 27
  25. 2019Continual Learning for Sentence Representations Using Conceptorscited by 22
  26. 2020An Empirical Investigation Towards Efficient Multi-Domain Language Model Pre-trainingcited by 7
  27. 2021COVID-19 Tweets Analysis through Transformer Language Modelscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 18; 2023: 30; 2024: 36; 2025: 18; 2026: 10.

114 papers.

Sort:

By area:

  1. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  2. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  3. 2025From RAG to Memory: Non-Parametric Continual Learning for Large Language Modelscited by 223
  4. 2023Continual Pre-training of Language Modelscited by 194
  5. 2023Mitigating the Alignment Tax of RLHFcited by 181
  6. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 179
  7. 2022Can Foundation Models Wrangle Your Data?cited by 173
  8. 2022Fine-tuned Language Models are Continual Learnerscited by 152
  9. 2024Continual Learning for Large Language Models: A Surveycited by 144
  10. 2023Continual Pre-Training of Large Language Models: How to (re)warm your model?cited by 135
  11. 2022TimeLMs: Diachronic Language Models from Twittercited by 129
  12. 2022TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Modelscited by 117
38 more
  1. 2024Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilitiescited by 111
  2. 2023Fine-tuning can cripple your foundation model; preserving features may be the solutioncited by 92
  3. 2021Time Waits for No One! Analysis and Challenges of Temporal Misalignmentcited by 91
  4. 2022Wild-Time: A Benchmark of in-the-Wild Distribution Shift over Timecited by 85
  5. 2022Continual Pre-Training Mitigates Forgetting in Language and Visioncited by 82
  6. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 78
  7. 2024Reliable, Adaptable, and Attributable Language Models with Retrievalcited by 73
  8. 2022ELLE: Efficient Lifelong Pre-training for Emerging Datacited by 70
  9. 2024Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Modelscited by 69
  10. 2023Can LMs Learn New Entities from Descriptions? Challenges in Propagating Injected Knowledgecited by 66
  11. 2024Towards Lifelong Learning of Large Language Models: A Surveycited by 60
  12. 2024Archon: An Architecture Search Framework for Inference-Time Techniquescited by 59
  13. 2024Investigating Continual Pretraining in Large Language Models: Insights and Implicationscited by 54
  14. 2024InsCL: A Data-efficient Continual Learning Paradigm for Fine-tuning Large Language Models with Instructionscited by 54
  15. 2025Injecting Domain-Specific Knowledge into Large Language Models: A Comprehensive Surveycited by 47
  16. 2024Recent Advances of Foundation Language Models-based Continual Learning: A Surveycited by 46
  17. 2024D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Modelscited by 39
  18. 2025Scaling Agents via Continual Pre-trainingcited by 39
  19. 2023Characterizing Learning Curves During Language Model Pre-Training: Learning, Forgetting, and Stabilitycited by 34
  20. 2023How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advancescited by 33
  21. 2023Efficient Continual Pre-training for Building Domain Specific Large Language Modelscited by 33
  22. 2024Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Modelscited by 29
  23. 2024SciLitLLM: How to Adapt LLMs for Scientific Literature Understandingcited by 29
  24. 2023Propagating Knowledge Updates to LMs Through Distillationcited by 28
  25. 2024Efficient Continual Pre-training by Mitigating the Stability Gapcited by 25
  26. 2024Set the Clock: Temporal Alignment of Pretrained Language Modelscited by 24
  27. 2023Goodtriever: Adaptive Toxicity Mitigation with Retrieval-augmented Modelscited by 23
  28. 2022On the Domain Adaptation and Generalization of Pretrained Language Models: A Surveycited by 22
  29. 2023An Overview on Language Models: Recent Developments and Outlookcited by 22
  30. 2024Continual LLaVA: Continual Instruction Tuning in Large Vision-Language Modelscited by 22
  31. 2023Dynamic Benchmarking of Masked Language Models on Temporal Concept Drift with Multiple Viewscited by 21
  32. 2023Mitigating Temporal Misalignment by Discarding Outdated Factscited by 21
  33. 2022TempoWiC: An Evaluation Benchmark for Detecting Meaning Shift in Social Mediacited by 19
  34. 2024NEO-BENCH: Evaluating Robustness of Large Language Models with Neologismscited by 19
  35. 2024Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oraclecited by 19
  36. 2023Recyclable Tuning for Continual Pre-trainingcited by 17
  37. 2023Carpe Diem: On the Evaluation of World Knowledge in Lifelong Language Modelscited by 17
  38. 2023ConPET: Continual Parameter-Efficient Tuning for Large Language Modelscited by 16

50 most cited of 114 All 114 (JSON)