# EvoLM: In Search of Lost Language Model Training Dynamics

arXiv: https://arxiv.org/abs/2506.16029 · HTML: https://mlsanity.com/paper/2506.16029/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Jun 2025.

## Built on by (most cited first, 16 of 16)

- [FlexGuard: Continuous Risk Scoring for Strictness-Adaptive LLM Content Moderation](https://mlsanity.com/paper/2602.23636/) (2026)
- [Theoretical Modeling of Large Language Model Self-Improvement Training Dynamics Through Solver-Verifier Gap](https://mlsanity.com/paper/2507.00075/) (2025)
- [Understanding Reasoning from Pretraining to Post-Training](https://mlsanity.com/paper/2607.16097/) (2026)
- [Full-bandwidth transformer](https://mlsanity.com/paper/2608.08888/) (2026)
- [Weight Decay Improves Language Model Plasticity](https://mlsanity.com/paper/2602.11137/) (2026)
- [Why Do Reasoning Models Lose Coverage? The Role of Data and Forks in the Road](https://mlsanity.com/paper/2605.17026/) (2026)
- [When RL Fails after SFT: Rejuvenating Model Plasticity for Robust SFT-to-RL Handoff](https://mlsanity.com/paper/2606.09932/) (2026)
- [When Can LLMs Learn to Reason with Weak Supervision?](https://mlsanity.com/paper/2604.18574/) (2026)
- [Sparsity Curse: Understanding RLVR Model Parameter Space from Model Merging](https://mlsanity.com/paper/2606.18521/) (2026)
- [Prescriptive Scaling Reveals the Evolution of Language Model Capabilities](https://mlsanity.com/paper/2602.15327/) (2026)
- [Scaling Reward Modeling without Human Supervision](https://mlsanity.com/paper/2603.02225/) (2026)
- [How Post-Training Shapes Biological Reasoning Models](https://mlsanity.com/paper/2606.16517/) (2026)
- [Ladder Up, Memory Down: Low-Cost Fine-Tuning With Side Nets](https://mlsanity.com/paper/2512.14237/) (2025)
- [Annotation Entropy Predicts Per-Example Learning Dynamics in LoRA Fine-Tuning](https://mlsanity.com/paper/2604.16332/) (2026)
- [PandaAI: A Practical Agent CQ2 for Neuro-symbolic Data Analysis And Integrated Decision-Making in Quantitative Finance](https://mlsanity.com/paper/2606.06823/) (2026)
- [SupraBench: A Benchmark for Supramolecular Chemistry](https://mlsanity.com/paper/2606.13477/) (2026)

## Abstract (arXiv)

Modern language model (LM) training has been divided into multiple stages, making it difficult for downstream developers to evaluate the impact of design choices made at each stage. We present EvoLM, a model suite that enables systematic and transparent analysis of LMs' training dynamics across pre-training, continued pre-training, supervised fine-tuning, and reinforcement learning. We train over 100 LMs with 1B and 4B parameters from scratch, and evaluate both upstream (language modeling) and downstream (problem-solving) capabilities, including considerations of both in-domain and out-of-domain generalization. Key insights highlight the diminishing returns from excessive pre-training and post-training, the importance and practices of mitigating forgetting during domain-specific continued pre-training, the crucial role of continued pre-training in bridging pre-training and post-training phases, and various intricate trade-offs when configuring supervised fine-tuning and reinforcement learning. To facilitate open research and reproducibility, we release all pre-trained and post-trained models, training datasets for all stages, and our entire training and evaluation pipeline.