mlsanityData to 19 Oct 2026

Continual pre-training and training dynamics

72 papers0 this weekHigh confidence in this grouping: the papers form one clear topicHigh confidence in this grouping: the papers form one clear topic · About

Keywords: continual, pre-training, language, knowledge, continual pre-training

Key facts. Continual pre-training and training dynamics: 72 papers on arXiv since 2020 (data through 19 Oct 2026). 32 of its 72 mapped papers (44%) cite “Don't Stop Pretraining: Adapt Language Models to Domains and Tasks” (2020, arXiv 2004.10964). 2 new papers 1 Oct–19 Oct.

What’s new

1 Oct – 19 Oct 2026

2 new since 1 Oct

  1. Fisher-Guided Submodular Data Selection for Continual Pre-Training of Large Language Models
  2. Persistent Depth Ordering amid Shifting Block-Bypass Responses in Language Model Pretraining

Foundations

Most cited by papers here

  1. Don't Stop Pretraining: Adapt Language Models to Domains and Tasks2020cited by 32 here
  2. Continual Pre-Training of Large Language Models: How to (re)warm your model?2023cited by 16 here
  3. Continual Pre-training of Language Models2023cited by 14 here
  4. Simple and Scalable Strategies to Continually Pre-train Large Language Models2024cited by 9 here
  5. Investigating Continual Pretraining in Large Language Models: Insights and Implications2024cited by 6 here

Lineage

The papers this direction builds on, oldest first.

Each paper on the line cites the one before.

  1. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 12 here
  2. 2021Deduplicating Training Data Makes Language Models Bettercited by 4 here
  3. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 4 here
  4. 2022Training Compute-Optimal Large Language Modelscited by 9 here
  5. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 12 here
  6. 2023Continual Pre-Training of Large Language Models: How to (re)warm your model?cited by 16 here
  7. 2024Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Modelscited by 6 here
  8. 2025TiC-LM: A Web-Scale Benchmark for Time-Continual LLM Pretrainingcited by 3 here
16 more it builds on, not in this chain
  1. 2020Scaling Laws for Neural Language Modelscited by 18 here
  2. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 32 here
  3. 2020An Empirical Investigation Towards Efficient Multi-Domain Language Model Pre-trainingcited by 2 here
  4. 2023Continual Pre-training of Language Modelscited by 14 here
  5. 2023Scaling Data-Constrained Language Modelscited by 3 here
  6. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 6 here
  7. 2023CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languagescited by 3 here
  8. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 8 here
  9. 2024OLMo: Accelerating the Science of Language Modelscited by 7 here
  10. 2024Investigating Continual Pretraining in Large Language Models: Insights and Implicationscited by 6 here
  11. 2024Simple and Scalable Strategies to Continually Pre-train Large Language Modelscited by 9 here
  12. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 5 here
  13. 2024D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Modelscited by 3 here
  14. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 3 here
  15. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 9 here
  16. 2024CMR Scaling Law: Predicting Critical Mixture Ratios for Continual Pre-training of Language Modelscited by 3 here

The 24 most cited of 72.

Browse

30 of 72, ranked 6–35

  1. 6Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Models20246
  2. 7TiC-LM: A Web-Scale Benchmark for Time-Continual LLM Pretraining20253
  3. 8D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Models20243
  4. 9Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilities20242
  5. 10Efficient Continual Pre-training by Mitigating the Stability Gap20242
  6. 11Efficient Continual Pre-training for Building Domain Specific Large Language Models20232
  7. 12An Empirical Investigation Towards Efficient Multi-Domain Language Model Pre-training20202
  8. 13When to Review: Spaced Repetition for Continual Pre-Training of Language Models20261
  9. 14PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runs20251
  10. 15Full-Parameter Continual Pretraining of Gemma2: Insights into Fluency and Domain Knowledge20251
All 30
  1. 16Improving Continual Pre-training Through Seamless Data Packing20251
  2. 17Breaking Language Barriers: Cross-Lingual Continual Pre-Training at Scale20241
  3. 18Velocitune: A Velocity-based Dynamic Domain Reweighting Method for Continual Pre-training20241
  4. 19Recyclable Tuning for Continual Pre-training20231
  5. 20Characterizing Learning Curves During Language Model Pre-Training: Learning, Forgetting, and Stability20231
  6. 21Pretraining Data Mixtures Enable Narrow Model Selection Capabilities in Transformer Models20231
  7. 22EcomGPT-CT: Continual Pre-training of E-commerce Large Language Models with Semi-structured Data20231
  8. 23Continued Pretraining for Better Zero- and Few-Shot Promptability20221
  9. 24How Do Large Language Models Learn Concepts During Continual Pre-Training?20260
  10. 25What Does Loss Optimization Actually Teach, If Anything? Knowledge Dynamics in Continual Pre-training of LLMs20260
  11. 26MAPLE: Metadata Conditioned LLM Pretraining for Locale-Aware Question Answering20260
  12. 27Tracing Persona Vectors Through LLM Pretraining20260
  13. 28TFGN: Task-Free, Replay-Free Continual Pre-Training Without Catastrophic Forgetting at LLM Scale20260
  14. 29Does Continued Pretraining on a Learner Corpus Improve Automated Essay Scoring on English Proficiency Tests? Evidence from EFCAMDAT20260
  15. 30When Probing Accuracy Saturates, Fragility Resolves: A Complementary Metric for LLM Pre-Training Analysis20260
  16. 31Emergent Capabilities Arise Randomly from Learning Sparse Attention Patterns20260
  17. 32Reading the News: Adapting Large Language Models to Swedish Journalism Through Continued Pre-Training20260
  18. 33Time-Incremental Continued Pretraining of LLMs: Knowledge Updates Without Catastrophic Forgetting20260
  19. 34Generalization Dynamics of LM Pre-training20260
  20. 35Replay on Demand: An Emergent Curriculum for Balancing Adaptation and Forgetting in Continued Pretraining20260