mlsanityData to 7 Oct 2026

Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Models

arXiv 2407.07263 ↗First posted 9 Jul 2024cs.CLcited by 69

Not on the map. Why?

Abstract (arXiv)

As language models have scaled both their number of parameters and pretraining dataset sizes, the computational cost for pretraining has become intractable except for the most well-resourced teams. This increasing cost makes it ever more important to be able to reuse a model after it has completed pretraining; allowing for a model's abilities to further improve without needing to train from scratch. In this work, we detail a set of guidelines that cover how to design efficacious data distributions and learning rate schedules for continued pretraining of language models. When applying these findings within a continued pretraining run on top of a well-trained 15B parameter model, we show an improvement of 9% in average model accuracy compared to the baseline of continued training on the pretraining set. The resulting recipe provides a practical starting point with which to begin developing language models through reuse rather than retraining.

Key facts. Cited by 69 papers in the map (data through 7 Oct 2026). Builds on 40 of its 50 references that are in the map. First posted on arXiv 9 Jul 2024.

Builds on

40 of 50 references.

References · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  10. 2017Billion-scale similarity search with GPUscited by 2,477
  11. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  12. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
28 more
  1. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  2. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  3. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  4. 2018Experience Replay for Continual Learningcited by 1,074
  5. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  6. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  7. 2024BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domainscited by 299
  8. 2023Physics of Language Models: Part 3.1, Knowledge Storage and Extractioncited by 264
  9. 2020Unsupervised Translation of Programming Languagescited by 262
  10. 2023Continual Pre-training of Language Modelscited by 194
  11. 2019CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEBcited by 177
  12. 2019SPoC: Search-based Pseudocode to Codecited by 176
  13. 2021Towards Continual Knowledge Learning of Language Modelscited by 172
  14. 2022Fine-tuned Language Models are Continual Learnerscited by 152
  15. 2019CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairscited by 136
  16. 2023Continual Pre-Training of Large Language Models: How to (re)warm your model?cited by 135
  17. 2022TimeLMs: Diachronic Language Models from Twittercited by 129
  18. 2022TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Modelscited by 117
  19. 2021Lifelong Pretraining: Continually Adapting Language Models to Emerging Corporacited by 114
  20. 2024LLaMA Pro: Progressive LLaMA with Block Expansioncited by 99
  21. 2022CERT: Continual Pre-Training on Sketches for Library-Oriented Code Generationcited by 87
  22. 2022ELLE: Efficient Lifelong Pre-training for Emerging Datacited by 70
  23. 2024Reka Core, Flash, and Edge: A Series of Powerful Multimodal Language Modelscited by 67
  24. 2024Does your data spark joy? Performance gains from domain upsampling at the end of trainingcited by 51
  25. 2020Online Fast Adaptation and Knowledge Accumulation: a New Approach to Continual Learningcited by 38
  26. 2021Understanding Continual Learning Settings with Data Distribution Drift Analysiscited by 38
  27. 2023Overcoming Catastrophic Forgetting in Massively Multilingual Continual Learningcited by 27
  28. 2024PLLaMa: An Open-source Large Language Model for Plant Sciencecited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 16; 2025: 33; 2026: 20.

69 papers.

Sort:

By area:

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  2. 2024Compact Language Models via Pruning and Knowledge Distillationcited by 142
  3. 2024LLM Pruning and Distillation in Practice: The Minitron Approachcited by 82
  4. 2024Synthetic continued pretrainingcited by 56
  5. 2025Canary-1B-v2 & Parakeet-TDT-0.6B-v3: Efficient and High-Performance Models for Multilingual ASR and ASTcited by 51
  6. 2024Upcycling Large Language Models into Mixture of Expertscited by 48
  7. 2025Scaling Agents via Continual Pre-trainingcited by 39
  8. 2025Patho-R1: A Multimodal Reinforcement Learning-Based Pathology Expert Reasonercited by 38
  9. 2024WildFeedback: Aligning LLMs With In-situ User Interactions And Feedbackcited by 31
  10. 2024Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Modelscited by 29
  11. 2024Scaling Law with Learning Rate Annealingcited by 29
  12. 2025Llama-3.1-FoundationAI-SecurityLLM-Base-8B Technical Reportcited by 28
38 more
  1. 2024Code-Switching Curriculum Learning for Multilingual Transfer in LLMscited by 26
  2. 2024A Practitioner's Guide to Continual Multimodal Pretrainingcited by 22
  3. 2025How Do LLMs Acquire New Knowledge? A Knowledge Circuits Perspective on Continual Pre-Trainingcited by 22
  4. 2024Task-Adaptive Pretrained Language Models via Clustered-Importance Samplingcited by 18
  5. 2025The Future of Continual Learning in the Era of Foundation Models: Three Key Directionscited by 16
  6. 2024MIND: Math Informed syNthetic Dialogues for Pretraining LLMscited by 14
  7. 2024Towards Effective and Efficient Continual Pre-training of Large Language Modelscited by 13
  8. 2025TiC-LM: A Web-Scale Benchmark for Time-Continual LLM Pretrainingcited by 13
  9. 2025Not All Parameters Are Created Equal: Smart Isolation Boosts Fine-Tuning Performancecited by 13
  10. 2025Memorization vs. Reasoning: Updating LLMs with New Knowledgecited by 12
  11. 2025One Tokenizer To Rule Them All: Emergent Language Plasticity via Multilingual Tokenizerscited by 11
  12. 2024BioMamba: Domain-Adaptive Biomedical Language Modelscited by 8
  13. 2025Domain Adaptation of Foundation LLMs for e-Commercecited by 8
  14. 2026OneReason Technical Reportcited by 8
  15. 2025Beyond Cosine Decay: On the effectiveness of Infinite Learning Rate Schedule for Continual Pre-trainingcited by 7
  16. 2026The Finetuner's Fallacy: When to Pretrain with Your Finetuning Datacited by 7
  17. 2025From Unaligned to Aligned: Scaling Multilingual LLMs with Multi-Way Parallel Corporacited by 6
  18. 2024LSHBloom: Memory-efficient, Extreme-scale Document Deduplicationcited by 5
  19. 2025LESA: Learnable LLM Layer Scaling-Upcited by 5
  20. 2025Continual Pre-training of MoEs: How robust is your router?cited by 5
  21. 2025Same accuracy, twice as fast: continuous training surpasses retraining from scratchcited by 4
  22. 2025Enhancing Domain-Specific Encoder Models with LLM-Generated Data: How to Leverage Ontologies, and How to Do Without Themcited by 4
  23. 2025Learning Dynamics in Continual Pre-Training for Large Language Modelscited by 4
  24. 2025Large-Scale Diverse Synthesis for Mid-Trainingcited by 3
  25. 2026Scaling Point-in-Time Language Modelscited by 3
  26. 2024Exploring Gen-AI applications in building research and industry: A reviewcited by 2
  27. 2024Train More Parameters But Mind Their Placement: Insights into Language Adaptation with PEFTcited by 2
  28. 2025NAACL2025 Tutorial: Adaptation of Large Language Modelscited by 2
  29. 2025Can Performant LLMs Be Ethical? Quantifying the Impact of Web Crawling Opt-Outscited by 2
  30. 2025Scaling Fine-Grained MoE Beyond 50B Parameters: Empirical Evaluation and Practical Insightscited by 2
  31. 2025Programming by Backprop: An Instruction is Worth 100 Examples When Finetuning LLMscited by 2
  32. 2025DIET-CP: Lightweight and Data Efficient Self Supervised Continued Pretrainingcited by 2
  33. 2026Parameter Importance is Not Static: Evolving Parameter Isolation for Supervised Fine-Tuningcited by 2
  34. 2025Domain-Adaptive Continued Pre-Training of Small Language Modelscited by 1
  35. 2025Improving Continual Pre-training Through Seamless Data Packingcited by 1
  36. 2025Minifinetuning: Low-Data Generation Domain Adaptation through Corrective Self-Distillationcited by 1
  37. 2025CultureMERT: Continual Pre-Training for Cross-Cultural Music Representation Learningcited by 1
  38. 2025Growing Transformers: Modular Composition and Layer-wise Expansion on a Frozen Substratecited by 1

50 most cited of 69 All 69 (JSON)