# Continual pre-training and training dynamics

Area: [LLM pre-training data and scaling laws](https://mlsanity.com/pretrain/index.md). HTML: https://mlsanity.com/pretrain/continual-pre-training-and-training-dynamics/

## Key facts

- Continual pre-training and training dynamics: 72 papers on arXiv since 2020 (data through 19 Oct 2026).
- 32 of its 72 mapped papers (44%) cite “Don't Stop Pretraining: Adapt Language Models to Domains and Tasks” (2020, arXiv 2004.10964).
- 16 of its 72 mapped papers (22%) cite “Continual Pre-Training of Large Language Models: How to (re)warm your model?” (2023, arXiv 2308.04014).
- 2 new papers 1 Oct–19 Oct.

## Most cited here

- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020), cited by 32 papers here
- [Continual Pre-Training of Large Language Models: How to (re)warm your model?](https://mlsanity.com/paper/2308.04014/) (2023), cited by 16 papers here
- [Continual Pre-training of Language Models](https://mlsanity.com/paper/2302.03241/) (2023), cited by 14 papers here
- [Simple and Scalable Strategies to Continually Pre-train Large Language Models](https://mlsanity.com/paper/2403.08763/) (2024), cited by 9 papers here
- [Investigating Continual Pretraining in Large Language Models: Insights and Implications](https://mlsanity.com/paper/2402.17400/) (2024), cited by 6 papers here
- [Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Models](https://mlsanity.com/paper/2407.07263/) (2024), cited by 6 papers here
- [D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Models](https://mlsanity.com/paper/2406.01375/) (2024), cited by 3 papers here
- [TiC-LM: A Web-Scale Benchmark for Time-Continual LLM Pretraining](https://mlsanity.com/paper/2504.02107/) (2025), cited by 3 papers here
- [An Empirical Investigation Towards Efficient Multi-Domain Language Model Pre-training](https://mlsanity.com/paper/2010.00784/) (2020), cited by 2 papers here
- [Efficient Continual Pre-training for Building Domain Specific Large Language Models](https://mlsanity.com/paper/2311.08545/) (2023), cited by 2 papers here

## New (1 Oct–19 Oct)

- [Fisher-Guided Submodular Data Selection for Continual Pre-Training of Large Language Models](https://mlsanity.com/paper/2610.02593/) (2026-10-01)
- [Persistent Depth Ordering amid Shifting Block-Bypass Responses in Language Model Pretraining](https://mlsanity.com/paper/2610.01165/) (2026-10-01)