mlsanityData to 7 Oct 2026

Continual Pre-Training of Large Language Models: How to (re)warm your model?

arXiv 2308.04014 ↗First posted 8 Aug 2023cs.CL cs.LGcited by 135

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) are routinely pre-trained on billions of tokens, only to restart the process over again once new data becomes available. A much cheaper and more efficient solution would be to enable the continual pre-training of these models, i.e. updating pre-trained models with new data instead of re-training them from scratch. However, the distribution shift induced by novel data typically results in degraded performance on past data. Taking a step towards efficient continual pre-training, in this work, we examine the effect of different warm-up strategies. Our hypothesis is that the learning rate must be re-increased to improve compute efficiency when training on a new dataset. We study the warmup phase of models pre-trained on the Pile (upstream data, 300B tokens) as we continue to pre-train on SlimPajama (downstream data, 297B tokens), following a linear warmup and cosine decay schedule. We conduct all experiments on the Pythia 410M language model architecture and evaluate performance through validation perplexity. We experiment with different pre-training checkpoints, various maximum learning rates, and various warmup lengths. Our results show that while rewarming models first increases the loss on upstream and downstream data, in the longer run it improves the downstream performance, outperforming models trained from scratchunicodex2013even for a large downstream dataset.

Key facts. Cited by 135 papers in the map (data through 7 Oct 2026). Builds on 44 of its 60 references that are in the map. First posted on arXiv 8 Aug 2023.

Builds on

44 of 60 references.

References · 44

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Segment Anythingcited by 8,158
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  6. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  7. 2023A Survey of Large Language Modelscited by 2,537
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  9. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  10. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  11. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  12. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
32 more
  1. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  2. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  3. 2019Orthogonal Gradient Descent for Continual Learningcited by 361
  4. 2017Variational Continual Learningcited by 335
  5. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
  6. 2021New Insights on Reducing Abrupt Representation Change in Online Continual Learningcited by 215
  7. 2023Continual Pre-training of Language Modelscited by 194
  8. 2020Understanding the Role of Training Regimes in Continual Learningcited by 177
  9. 2019BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysiscited by 176
  10. 2021Towards Continual Knowledge Learning of Language Modelscited by 172
  11. 2022Fine-tuned Language Models are Continual Learnerscited by 152
  12. 2021Self-Supervised Models are Continual Learnerscited by 149
  13. 2019Continual Unsupervised Representation Learningcited by 147
  14. 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 130
  15. 2022TimeLMs: Diachronic Language Models from Twittercited by 129
  16. 2022TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Modelscited by 117
  17. 2021Lifelong Pretraining: Continually Adapting Language Models to Emerging Corporacited by 114
  18. 2021Representational Continuity for Unsupervised Continual Learningcited by 95
  19. 2019Lifelong GAN: Continual Learning for Conditional Image Generationcited by 78
  20. 2022Probing Representation Forgetting in Supervised and Unsupervised Continual Learningcited by 74
  21. 2022ELLE: Efficient Lifelong Pre-training for Emerging Datacited by 70
  22. 2018Generative Models from the perspective of Continual Learningcited by 69
  23. 2022Continual Learning with Foundation Models: An Empirical Study of Latent Replaycited by 61
  24. 2022Continual evaluation for lifelong learning: Identifying the stability gapcited by 55
  25. 2017Continual Learning in Generative Adversarial Netscited by 51
  26. 2020ECONET: Effective Continual Pretraining of Language Models for Event Temporal Reasoningcited by 38
  27. 2021Understanding Continual Learning Settings with Data Distribution Drift Analysiscited by 38
  28. 2019IMHO Fine-Tuning Improves Claim Detectioncited by 27
  29. 2023Overcoming Catastrophic Forgetting in Massively Multilingual Continual Learningcited by 27
  30. 2023Adapting a Language Model While Preserving its General Knowledgecited by 23
  31. 2021On Anytime Learning at Macroscalecited by 20
  32. 2022Challenging Common Assumptions about Catastrophic Forgettingcited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 11; 2024: 55; 2025: 35; 2026: 34.

135 papers.

Sort:

By area:

  1. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  2. 2023MEDITRON-70B: Scaling Medical Pretraining for Large Language Modelscited by 272
  3. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  4. 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 160
  5. 2024Simple and Scalable Strategies to Continually Pre-train Large Language Modelscited by 122
  6. 2024Zamba: A Compact 7B SSM Hybrid Modelcited by 118
  7. 2023DiLoCo: Distributed Low-Communication Training of Language Modelscited by 115
  8. 2024Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilitiescited by 111
  9. 2024LLaMA Pro: Progressive LLaMA with Block Expansioncited by 99
  10. 2024Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Modelscited by 69
  11. 2024Towards Lifelong Learning of Large Language Models: A Surveycited by 60
  12. 2024Synthetic continued pretrainingcited by 56
38 more
  1. 2024Investigating Continual Pretraining in Large Language Models: Insights and Implicationscited by 54
  2. 2024LLaMAX: Scaling Linguistic Horizons of LLM by Enhancing Translation Capabilities Beyond 100 Languagescited by 47
  3. 2024CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuningcited by 41
  4. 2024D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Modelscited by 39
  5. 2025Scaling Agents via Continual Pre-trainingcited by 39
  6. 2023OFA: A Framework of Initializing Unseen Subword Embeddings for Efficient Large-scale Multilingual Continued Pretrainingcited by 37
  7. 2023How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advancescited by 33
  8. 2025SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuningcited by 32
  9. 2024Scaling Law with Learning Rate Annealingcited by 29
  10. 2024SciLitLLM: How to Adapt LLMs for Scientific Literature Understandingcited by 29
  11. 2024Parallel Structures in Pre-training Data Yield In-Context Learningcited by 26
  12. 2024Efficient Continual Pre-training by Mitigating the Stability Gapcited by 25
  13. 2024Tele-LLMs: A Series of Specialized Large Language Models for Telecommunicationscited by 25
  14. 2023LEMON: Lossless model expansioncited by 23
  15. 2025A Survey on Agentic Multimodal Large Language Modelscited by 19
  16. 2024Salute the Classic: Revisiting Challenges of Machine Translation in the Age of Large Language Modelscited by 17
  17. 2024Examining Forgetting in Continual Pre-training of Aligned Large Language Modelscited by 16
  18. 2024Breaking Language Barriers: Cross-Lingual Continual Pre-Training at Scalecited by 15
  19. 2023Stabilizing RLHF through Advantage Model and Selective Rehearsalcited by 14
  20. 2024Domain Adaptation of Llama3-70B-Instruct through Continual Pre-Training and Model Merging: A Comprehensive Evaluationcited by 14
  21. 2024Banishing LLM Hallucinations Requires Rethinking Generalizationcited by 13
  22. 2024Towards Effective and Efficient Continual Pre-training of Large Language Modelscited by 13
  23. 2025TiC-LM: A Web-Scale Benchmark for Time-Continual LLM Pretrainingcited by 13
  24. 2025Revisiting Replay and Gradient Alignment for Continual Pre-Training of Large Language Modelscited by 13
  25. 2026AfriqueLLM: How Data Mixing and Model Architecture Impact Continued Pre-training for African Languagescited by 12
  26. 2026Replaying pre-training data improves fine-tuningcited by 12
  27. 2024BBox-Adapter: Lightweight Adapting for Black-Box Large Language Modelscited by 11
  28. 2024SambaLingo: Teaching Large Language Models New Languagescited by 11
  29. 2024Scalable Data Ablation Approximations for Language Models through Modular Training and Mergingcited by 11
  30. 2024Aurora-M: Open Source Continual Pre-training for Multilingual Language and Codecited by 10
  31. 2024SLCA++: Unleash the Power of Sequential Fine-tuning for Continual Learning with Pre-trainingcited by 10
  32. 2023EcomGPT-CT: Continual Pre-training of E-commerce Large Language Models with Semi-structured Datacited by 9
  33. 2024Exploring Forgetting in Large Language Model Pre-Trainingcited by 9
  34. 2025Training Dynamics of the Cooldown Stage in Warmup-Stable-Decay Learning Rate Schedulercited by 9
  35. 2024BayLing 2: A Multilingual Large Language Model with Efficient Language Alignmentcited by 8
  36. 2025Domain Adaptation of Foundation LLMs for e-Commercecited by 8
  37. 2025EnronQA: Towards Personalized RAG over Private Documentscited by 8
  38. 2026Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averagingcited by 8

50 most cited of 135 All 135 (JSON)