mlsanityData to 14 Oct 2026

Replaying pre-training data improves fine-tuning

arXiv 2603.04964 ↗First posted 5 Mar 2026cs.CL cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

To obtain a language model for a target domain (e.g. math), the current paradigm is to pre-train on a vast amount of generic web text and then fine-tune on the relatively limited amount of target data. Typically, generic data is only mixed in during fine-tuning to prevent catastrophic forgetting of the generic domain. We surprisingly find that replaying the generic data during fine-tuning can actually improve performance on the (less related) target task. Concretely, in a controlled pre-training environment with 4M target tokens, 4B total tokens, and 150M parameter models, generic replay increases target data efficiency by up to 1.87× for fine-tuning and 2.06× for mid-training. We further analyze data schedules that introduce target data during pre-training and find that replay helps more when there is less target data present in pre-training. We demonstrate the success of replay in practice for fine-tuning 8B parameter models, improving agentic web navigation success by 4.5% and Basque question-answering accuracy by 2%.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 52 of its 69 references that are in the map. First posted on arXiv 5 Mar 2026.

Builds on

52 of 69 references.

References · 52

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  4. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  5. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,330
  6. 2022Editing Models with Task Arithmeticcited by 1,120
  7. 2015Prioritized Experience Replaycited by 1,099
  8. 2018Experience Replay for Continual Learningcited by 1,095
  9. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  10. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  11. 2021Robust fine-tuning of zero-shot modelscited by 797
  12. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
38 more
  1. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  2. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 513
  3. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  4. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 399
  5. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 333
  6. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 288
  7. 2021Scaling Laws for Transfercited by 272
  8. 2023Physics of Language Models: Part 3.1, Knowledge Storage and Extractioncited by 270
  9. 2020Revisiting Few-sample BERT Fine-tuningcited by 238
  10. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 236
  11. 2024Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?cited by 232
  12. 2018Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Taskscited by 215
  13. 2024WebLINX: Real-World Website Navigation with Multi-Turn Dialoguecited by 180
  14. 2023Understanding Catastrophic Forgetting in Language Models via Implicit Inferencecited by 144
  15. 2023Continual Pre-Training of Large Language Models: How to (re)warm your model?cited by 137
  16. 2024Simple and Scalable Strategies to Continually Pre-train Large Language Modelscited by 127
  17. 2023Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Modelscited by 121
  18. 2023A Spectral Condition for Feature Learningcited by 109
  19. 2020When Do Curricula Work?cited by 106
  20. 2024How Do Large Language Models Acquire Factual Knowledge During Pretraining?cited by 90
  21. 2024Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspectivecited by 81
  22. 2024Scaling Laws for Data Filtering -- Data Curation cannot be Compute Agnosticcited by 77
  23. 2024Scaling Exponents Across Parameterizations and Optimizerscited by 76
  24. 2025Overtrained Language Models Are Harder to Fine-Tunecited by 58
  25. 2024Investigating Continual Pretraining in Large Language Models: Insights and Implicationscited by 57
  26. 2024Instruction-tuned Language Models are Better Knowledge Learnerscited by 55
  27. 2024Does your data spark joy? Performance gains from domain upsampling at the end of trainingcited by 52
  28. 2024Scaling Laws for Predicting Downstream Performance in LLMscited by 48
  29. 2024Adaptive Data Optimization: Dynamic Sample Selection with Scaling Lawscited by 47
  30. 2024Improving Pretraining Data Using Perplexity Correlationscited by 46
  31. 2023Optimal Linear Decay Learning Rate Schedules and Further Refinementscited by 43
  32. 2024Latxa: An Open Language Model and Evaluation Suite for Basquecited by 42
  33. 2025Weight Ensembling Improves Reasoning in Language Modelscited by 38
  34. 2025The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling for Large Model Trainingcited by 33
  35. 2024Understanding Finetuning for Factual Knowledge Extractioncited by 32
  36. 2023Provable Advantage of Curriculum Learning on Parity Targets with Mixed Inputscited by 22
  37. 2025Not All Tokens Are What You Need In Thinkingcited by 20
  38. 2023Irreducible Curriculum for Language Model Pretrainingcited by 16

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 13.

13 papers.

Sort:

By area:

  1. 2026Sharpness-Aware Pretraining Mitigates Catastrophic Forgettingcited by 10
  2. 2026The Finetuner's Fallacy: When to Pretrain with Your Finetuning Datacited by 9
  3. 2026Data-efficient pre-training by scaling synthetic megadocscited by 7
  4. 2026Forgetting in Language Models: Capacity, Optimization, and Self-Generated Replaycited by 7
  5. 2026Alignment Whack-a-Mole : Finetuning Activates Verbatim Recall of Copyrighted Books in Large Language Modelscited by 6
  6. 2026(How) Learning Rates Regulate Catastrophic Overtrainingcited by 6
  7. 2026Early Data Exposure Improves Robustness to Subsequent Fine-Tuningcited by 4
  8. 2026ufakzeka-1: Building and Evaluating a 151M-Parameter Turkish Language Model from Scratchcited by 2
  9. 2026Synthetic Mixed Training: Scaling Parametric Knowledge Acquisition Beyond RAGcited by 1
  10. 2026Continual Learning via Self-Probe Gradientscited by 1
  11. 2026Loss Smoothing for Stable Adaptation Under Distribution Shiftcited by 0
  12. 2026Thinking at the Right Size: Amortized Distillation Across Post-Trained LLMscited by 0
1 more
  1. 2026Popular Knowledge Propagates More Errors in LLM Knowledge Updatingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Why Reinforcement Fine-Tuning Enables MLLMs Preserve Prior Knowledge Better: A Data Perspective
  2. 2026On-Policy Replay for Continual Supervised Fine-Tuning
  3. 2025Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replay
  4. 2026Are Parameter-Efficient Fine-tuning Methods Really Different?
  5. 2025Balancing Synthetic Data and Replay for Enhancing Task-Specific Capabilities
  6. 2025Contextual Experience Replay for Self-Improvement of Language Agents
  7. 2026Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Models
  8. 2026Data Repetition Beats Data Scaling in Long-CoT Supervised Fine-Tuning