# Revisiting Replay and Gradient Alignment for Continual Pre-Training of Large Language Models

arXiv: https://arxiv.org/abs/2508.01908 · HTML: https://mlsanity.com/paper/2508.01908/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 62 references that are in the map.
- First posted on arXiv 3 Aug 2025.

## Builds on (most cited first, 30 of 38)

- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [Efficient Lifelong Learning with A-GEM](https://mlsanity.com/paper/1812.00420/) (2018)
- [On First-Order Meta-Learning Algorithms](https://mlsanity.com/paper/1803.02999/) (2018)
- [Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interference](https://mlsanity.com/paper/1810.11910/) (2018)
- [PathNet: Evolution Channels Gradient Descent in Super Neural Networks](https://mlsanity.com/paper/1701.08734/) (2017)
- [Lookahead Optimizer: k steps forward, 1 step back](https://mlsanity.com/paper/1907.08610/) (2019)
- [Towards Continual Reinforcement Learning: A Review and Perspectives](https://mlsanity.com/paper/2012.13490/) (2020)
- [Continual Learning of Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2404.16789/) (2024)
- [Continual Pre-training of Language Models](https://mlsanity.com/paper/2302.03241/) (2023)
- [Continual Learning with Pre-Trained Models: A Survey](https://mlsanity.com/paper/2401.16386/) (2024)
- [Towards a Cleaner Document-Oriented Multilingual Crawled Corpus](https://mlsanity.com/paper/2201.06642/) (2022)
- [Continual Pre-Training of Large Language Models: How to (re)warm your model?](https://mlsanity.com/paper/2308.04014/) (2023)
- [DEMix Layers: Disentangling Domains for Modular Language Modeling](https://mlsanity.com/paper/2108.05036/) (2021)
- [A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages](https://mlsanity.com/paper/2006.06202/) (2020)
- [Simple and Scalable Strategies to Continually Pre-train Large Language Models](https://mlsanity.com/paper/2403.08763/) (2024)
- [Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learning](https://mlsanity.com/paper/1711.01239/) (2017)
- [Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora](https://mlsanity.com/paper/2110.08534/) (2021)
- [Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilities](https://mlsanity.com/paper/2404.17790/) (2024)
- [Lifelong Language Pretraining with Distribution-Specialized Experts](https://mlsanity.com/paper/2305.12281/) (2023)
- [Routing Networks and the Challenges of Modular and Compositional Computation](https://mlsanity.com/paper/1904.12774/) (2019)
- [Low-Rank Approximations for Conditional Feedforward Computation in Deep Neural Networks](https://mlsanity.com/paper/1312.4461/) (2013)
- [Automatically Composing Representation Transformations as a Means for Generalization](https://mlsanity.com/paper/1807.04640/) (2018)
- [Scalable Recollections for Continual Lifelong Learning](https://mlsanity.com/paper/1711.06761/) (2017)
- [A Practitioner's Guide to Continual Multimodal Pretraining](https://mlsanity.com/paper/2408.14471/) (2024)

## Built on by (most cited first, 14 of 14)

- [CRL-VLA: Continual Vision-Language-Action Learning](https://mlsanity.com/paper/2602.03445/) (2026)
- [Harness Continual Learning: Continual Adaptation Beyond Model Parameters](https://mlsanity.com/paper/2608.19013/) (2026)
- [Beyond Sliding Windows: Learning to Manage Memory in Non-Markovian Environments](https://mlsanity.com/paper/2512.19154/) (2025)
- [Training Data Selection with Gradient Orthogonality for Efficient Domain Adaptation](https://mlsanity.com/paper/2602.06359/) (2026)
- [The Effectiveness of Approximate Regularized Replay for Efficient Supervised Fine-Tuning of Large Language Models](https://mlsanity.com/paper/2512.22337/) (2025)
- [Attribution-Guided Continual Learning for Large Language Models](https://mlsanity.com/paper/2605.05285/) (2026)
- [Position: Collusion Risks Among AI Reasoning Agents Justify Certification Requirements for Making Market Decisions](https://mlsanity.com/paper/2608.18078/) (2026)
- [Replay-Based Continual Learning for Physics-Informed Neural Operators](https://mlsanity.com/paper/2605.04832/) (2026)
- [TFGN: Task-Free, Replay-Free Continual Pre-Training Without Catastrophic Forgetting at LLM Scale](https://mlsanity.com/paper/2605.15053/) (2026)
- [Fast and Slow Variational Continual Learning](https://mlsanity.com/paper/2606.24007/) (2026)
- [Relative Parameter Importance in Task-Agnostic Replay-Free Continual Learning](https://mlsanity.com/paper/2608.00630/) (2026)
- [Continual Learning in Transition](https://mlsanity.com/paper/2608.06216/) (2026)
- [MASCRDM: Multi-Agent System for Compliance Risk Detection and Mitigation in Training Process of Large Language Models](https://mlsanity.com/paper/2609.39107/) (2026)
- [Replay on Demand: An Emergent Curriculum for Balancing Adaptation and Forgetting in Continued Pretraining](https://mlsanity.com/paper/2609.40089/) (2026)

## Abstract (arXiv)

Training large language models (LLMs) typically involves pre-training on massive corpora, only to restart the process entirely when new data becomes available. A more efficient and resource-conserving approach would be continual pre-training, where models are updated with new data rather than retraining from scratch. However, the introduction of new data often causes distribution shifts, leading to performance degradation on previously learned tasks. In this paper, we take a deeper look at two popular proposals for addressing this distribution shift within the continual learning literature: experience replay and gradient alignment. We consider continual pre-training of models within the Llama family of architectures at a large scale across languages with 100 billion tokens of training data in each language, finding that both replay and gradient alignment lead to more stable learning without forgetting. This conclusion holds both as we vary the model scale and as we vary the number and diversity of tasks. Moreover, we are the first to demonstrate the effectiveness of gradient alignment techniques in the context of LLM pre-training and propose an efficient implementation of meta-experience replay (MER) that imbues experience replay with the benefits of gradient alignment despite negligible compute and memory overhead. Our scaling analysis across model sizes and replay rates indicates that small rates of replaying old examples are definitely a more valuable use of compute than investing in model size, but that it is more compute efficient to scale the size of the model than invest in high rates of replaying old examples.