mlsanityData to 7 Oct 2026

Overtrained Language Models Are Harder to Fine-Tune

arXiv 2503.19206 ↗First posted 24 Mar 2025cs.CL cs.AIcited by 57

Not on the map. Why?

Abstract (arXiv)

Large language models are pre-trained on ever-growing token budgets under the assumption that better pre-training performance translates to improved downstream models. In this work, we challenge this assumption and show that extended pre-training can make models harder to fine-tune, leading to degraded final performance. We term this phenomenon catastrophic overtraining. For example, the instruction-tuned OLMo-1B model pre-trained on 3T tokens leads to over 2% worse performance on multiple standard LLM benchmarks than its 2.3T token counterpart. Through controlled experiments and theoretical analysis, we show that catastrophic overtraining arises from a systematic increase in the broad sensitivity of pre-trained parameters to modifications, including but not limited to fine-tuning. Our findings call for a critical reassessment of pre-training design that considers the downstream adaptability of the model.

Key facts. Cited by 57 papers in the map (data through 7 Oct 2026). First posted on arXiv 24 Mar 2025.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 29; 2026: 28.

57 papers.

Sort:

By area:

  1. 2025Tracing the Representation Geometry of Language Models from Pretraining to Post-trainingcited by 33
  2. 2025RL Fine-Tuning Heals OOD Forgetting in SFTcited by 27
  3. 2025Pre-training under infinite computecited by 24
  4. 2025Mid-Training of Large Language Models: A Surveycited by 18
  5. 2025Unveiling the Basin-Like Loss Landscape in Large Language Modelscited by 13
  6. 2025Activation Control for Efficiently Eliciting Long Chain-of-thought Ability of Language Modelscited by 12
  7. 2026Replaying pre-training data improves fine-tuningcited by 12
  8. 2026Pre-training LLM without Learning Rate Decay Enhances Supervised Fine-Tuningcited by 12
  9. 2025Midtraining Bridges Pretraining and Posttraining Distributionscited by 10
  10. 2025The Butterfly Effect: Neural Network Training Trajectories Are Highly Sensitive to Initial Conditionscited by 9
  11. 2025The wall confronting large language modelscited by 8
  12. 2025Scaling with Collapse: Efficient and Predictable Training of LLM Familiescited by 8
38 more
  1. 2025Can You Learn to See Without Images? Procedural Warm-Up for Vision Transformerscited by 8
  2. 2025Posterior Behavioral Cloning: Pretraining BC Policies for Efficient RL Finetuningcited by 8
  3. 2025LLM Fingerprinting via Semantically Conditioned Watermarkscited by 7
  4. 2026Sharpness-Aware Pretraining Mitigates Catastrophic Forgettingcited by 7
  5. 2025Reinitializing weights vs units for maintaining plasticity in neural networkscited by 6
  6. 2025Towards Active Synthetic Data Generation for Finetuning Language Modelscited by 6
  7. 2025Overcoming Multi-step Complexity in Multimodal Theory-of-Mind Reasoning: A Scalable Bayesian Plannercited by 5
  8. 2025A Theory of Inference Compute Scaling: Reasoning through Directed Stochastic Skill Searchcited by 5
  9. 2025Universal Properties of Activation Sparsity in Modern Large Language Modelscited by 5
  10. 2026Forgetting in Language Models: Capacity, Optimization, and Self-Generated Replaycited by 5
  11. 2025Massively Multilingual Adaptation of Large Language Models Using Bilingual Translation Datacited by 4
  12. 2025Conda: Column-Normalized Adam for Training Large Language Models Fastercited by 4
  13. 2026What Do Agents Learn from Trajectory-SFT: Semantics or Interfaces?cited by 4
  14. 2026The Effect of Mini-Batch Noise on the Implicit Bias of Adamcited by 4
  15. 2026Nexus: Same Pretraining Loss, Better Downstream Generalization via Common Minimacited by 4
  16. 2026(How) Learning Rates Regulate Catastrophic Overtrainingcited by 4
  17. 2025Finding the Minimal Parameter Budget for Implicit Reasoning: A Data Complexity Driven Scaling Law for Language Modelscited by 3
  18. 2025Can Generalist Vision Language Models (VLMs) Rival Specialist Medical VLMs? Benchmarking and Strategic Insightscited by 3
  19. 2025xLSTM Scaling Laws: Competitive Performance with Linear Time-Complexitycited by 3
  20. 2026Towards Agentic Intelligence for Materials Sciencecited by 3
  21. 2026Test-Time Scaling Makes Overtraining Compute-Optimalcited by 3
  22. 2026Optimizer-Model Consistency: Full Finetuning with the Same Optimizer as Pretraining Forgets Lesscited by 3
  23. 2026TMRL: Diffusion Timestep-Modulated Pretraining Enables Exploration for Efficient Policy Finetuningcited by 3
  24. 2026Early Data Exposure Improves Robustness to Subsequent Fine-Tuningcited by 3
  25. 2026Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectorscited by 3
  26. 2025From Memorization to Parameter Interference: How Overtraining Experts Harms Model Mergingcited by 2
  27. 2025Reusing Overtrained Language Models Saturates Scalingcited by 2
  28. 2026When pre-training hurts LoRA fine-tuning: a dynamical analysis via single-index modelscited by 2
  29. 2026Disentangling Geometry, Performance, and Training in Language Modelscited by 2
  30. 2026Divergence Decoding: Inference-Time Unlearning via Auxiliary Modelscited by 2
  31. 2026TailSFT: Filtered Fine-Tuning Improves Post-Training Performancecited by 2
  32. 2025GaLLoP: Gradient-based Sparse Learning on Low-Magnitude Parameterscited by 1
  33. 2026Optimal Splitting of Language Models from Mixtures to Specialized Domainscited by 1
  34. 2026Can Scale Save Us From Plasticity Loss in Large Language Models?cited by 1
  35. 2026Manifold-Constrained Hyper-Connections for Parameter-Efficient Finetuningcited by 1
  36. 2025Evaluation of Finetuned LLMs in AMR Parsingcited by 0
  37. 2025Predicting Training Re-evaluation Curves Enables Effective Data Curriculums for LLMscited by 0
  38. 2025PTPP-Aware Adaptation Scaling Laws: Predicting Domain-Adaptation Performance at Unseen Pre-Training Budgetscited by 0

50 most cited of 57 All 57 (JSON)