mlsanityData to 14 Oct 2026

Scaling Laws for Forgetting during Finetuning with Pretraining Data Injection

arXiv 2502.06042 ↗First posted 9 Feb 2025cs.LG cs.CLcited by 35

Not on the map. Why?

Abstract (arXiv)

A widespread strategy to obtain a language model that performs well on a target domain is to finetune a pretrained model to perform unsupervised next-token prediction on data from that target domain. Finetuning presents two challenges: (i) if the amount of target data is limited, as in most practical applications, the model will quickly overfit, and (ii) the model will drift away from the original model, forgetting the pretraining data and the generic knowledge that comes with it. We aim to derive scaling laws that quantify these two phenomena for various target domains, amounts of available target data, and model scales. We measure the efficiency of injecting pretraining data into the finetuning data mixture to avoid forgetting and mitigate overfitting. A key practical takeaway from our study is that injecting as little as 1% of pretraining data in the finetuning data mixture prevents the model from forgetting the pretraining set.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Builds on 35 of its 44 references that are in the map. First posted on arXiv 9 Feb 2025.

Builds on

35 of 44 references.

References · 35

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  8. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  9. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  10. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  11. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  12. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
23 more
  1. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  2. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  3. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  4. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 639
  5. 2017Deep Learning Scaling is Predictable, Empiricallycited by 582
  6. 2023Scaling Data-Constrained Language Modelscited by 403
  7. 2021Scaling Laws for Transfercited by 272
  8. 2024When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Methodcited by 204
  9. 2024RedPajama: an Open Dataset for Training Large Language Modelscited by 182
  10. 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 170
  11. 2021Self-attention Does Not Need O(n2) Memorycited by 154
  12. 2023Scaling Laws for Generative Mixed-Modal Language Modelscited by 134
  13. 2024Simple and Scalable Strategies to Continually Pre-train Large Language Modelscited by 127
  14. 2024Resolving Discrepancies in Compute-Optimal Scaling of Language Modelscited by 88
  15. 2024Asymmetry in Low-Rank Adapters of Foundation Modelscited by 86
  16. 2024Chinchilla Scaling: A replication attemptcited by 72
  17. 2024Scaling Laws for Forgetting When Fine-Tuning Large Language Modelscited by 38
  18. 2021Improved Fine-Tuning by Better Leveraging Pre-Training Datacited by 36
  19. 2024Get more for less: Principled Data Selection for Warming Up Fine-Tuning in LLMscited by 32
  20. 2024Scaling Law with Learning Rate Annealingcited by 29
  21. 2024Disentangling and Mitigating the Impact of Task Similarity for Continual Learningcited by 19
  22. 2024When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Trainingcited by 15
  23. 2024An Empirical Study of Scaling Laws for Transfercited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 11; 2026: 24.

35 papers.

Sort:

By area:

  1. 2025Scaling Laws for Optimal Data Mixturescited by 52
  2. 2025Enhancing Multilingual LLM Pretraining with Model-Based Data Selectioncited by 35
  3. 2025Closing the Gap Between Text and Speech Understanding in LLMscited by 17
  4. 2026Counteraction-Aware Multi-Teacher On-Policy Distillation for General Capability Recovery with Domain Preservationcited by 15
  5. 2025Midtraining Bridges Pretraining and Posttraining Distributionscited by 12
  6. 2026DatedGPT: Preventing Lookahead Bias in Large Language Models with Time-Aware Pretrainingcited by 8
  7. 2026Forgetting in Language Models: Capacity, Optimization, and Self-Generated Replaycited by 7
  8. 2026Scaling Laws for Mixture Pretraining Under Data Constraintscited by 6
  9. 2025Breaking Language Barriers in Visual Language Models via Multilingual Textual Regularizationcited by 4
  10. 2025Mining Useful General Data for Low-Resource Domain Adaptationcited by 4
  11. 2026Diversity in Large Language Models under Supervised Fine-Tuningcited by 4
  12. 2026DynaMiCS: Fine-tuning LLMs with Performance Constraints using Dynamic Mixturescited by 4
23 more
  1. 2026Early Data Exposure Improves Robustness to Subsequent Fine-Tuningcited by 4
  2. 2026TACTICL: Task-Aware Compression of Tabular ICL Modelscited by 4
  3. 2026Causal Data Augmentation for Robust Fine-Tuning of Tabular Foundation Modelscited by 3
  4. 2025Reusing Overtrained Language Models Saturates Scalingcited by 2
  5. 2025From Amateur to Master: Infusing Knowledge into LLMs via Automated Curriculum Learningcited by 2
  6. 2026The Culture Funnel: You Can't Align What isn't in the Datacited by 2
  7. 2026Scaling Laws for Hypernetwork-Based Knowledge Injection in Large Language Modelscited by 2
  8. 2025SUIT: Knowledge Editing with Subspace-Aware Key-Value Mappingscited by 1
  9. 2026Optimal Splitting of Language Models from Mixtures to Specialized Domainscited by 1
  10. 2026The Future of Facts: Tracing the Factual Generation-Verification Gapcited by 1
  11. 2026Knowledge Editing for Masked Diffusion Language Modelscited by 1
  12. 2026Continual Learning via Self-Probe Gradientscited by 1
  13. 2025PTPP-Aware Adaptation Scaling Laws: Predicting Domain-Adaptation Performance at Unseen Pre-Training Budgetscited by 0
  14. 2025Magnitude-Modulated Equivariant Adapter for Parameter-Efficient Fine-Tuning of Equivariant Graph Neural Networkscited by 0
  15. 2026Awakening the Sleeping Agent: Lean-Specific Agentic Data Reactivates General Tool Use in Goedel Provercited by 0
  16. 2026Teaching LLMs Brazilian Healthcare: Injecting Knowledge from Official Clinical Guidelinescited by 0
  17. 2026Always Learning, Always Mixing: Efficient and Simple Data Mixing All The Timecited by 0
  18. 2026A mathematical theory of balancing relational generalization and memorizationcited by 0
  19. 2026KARMA: Knowledge graph-based Automated Reasoning Materialization and Alignmentcited by 0
  20. 2026A Predict-then-Correct Loop Based on Few-Shot Continuous Contextual Bandit for Demand Forecastingcited by 0
  21. 2026Recovering General Capabilities via Uncertainty-Calibrated Multi-Teacher On-Policy Distillationcited by 0
  22. 2026CoLT-Drive: Counterfactual Long-Tail Benchmarking and Knowledge-Preserving Adaptation for Driving Affordance Predictioncited by 0
  23. 2026SAMBAR: Selective Anchoring via Method of Multipliers for Balanced Knowledge Acquisition and Retention in Vision-Language-Action Modelscited by 0