# Preservation Is Not Enough for Width Growth: Regime-Sensitive Selection of Dense LM Warm Starts

arXiv: https://arxiv.org/abs/2604.04281 · HTML: https://mlsanity.com/paper/2604.04281/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 17 references that are in the map.
- First posted on arXiv 5 Apr 2026.

## Builds on (most cited first, 16 of 16)

- [TinyStories: How Small Can Language Models Be and Still Speak Coherent English?](https://mlsanity.com/paper/2305.07759/) (2023)
- [Net2Net: Accelerating Learning via Knowledge Transfer](https://mlsanity.com/paper/1511.05641/) (2015)
- [Small-scale proxies for large-scale Transformer training instabilities](https://mlsanity.com/paper/2309.14322/) (2023)
- [Learning to Grow Pretrained Models for Efficient Transformer Training](https://mlsanity.com/paper/2303.00980/) (2023)
- [Cramming: Training a Language Model on a Single GPU in One Day](https://mlsanity.com/paper/2212.14034/) (2022)
- [bert2BERT: Towards Reusable Pretrained Language Models](https://mlsanity.com/paper/2110.07143/) (2021)
- [Network Morphism](https://mlsanity.com/paper/1603.01670/) (2016)
- [Staged Training for Transformer Language Models](https://mlsanity.com/paper/2203.06211/) (2022)
- [Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Training](https://mlsanity.com/paper/2405.15319/) (2024)
- [Scaling Smart: Accelerating Large Language Model Pre-training with Small Model Initialization](https://mlsanity.com/paper/2409.12903/) (2024)
- [LEMON: Lossless model expansion](https://mlsanity.com/paper/2310.07999/) (2023)
- [Efficient Construction of Model Family through Progressive Training Using Model Expansion](https://mlsanity.com/paper/2504.00623/) (2025)
- [Landscape-Aware Growing: The Power of a Little LAG](https://mlsanity.com/paper/2406.02469/) (2024)
- [Warmstarting for Scaling Language Models](https://mlsanity.com/paper/2411.07340/) (2024)
- [SPARKLING: Balancing Signal Preservation and Symmetry Breaking for Width-Progressive Learning](https://mlsanity.com/paper/2602.02472/) (2026)
- [Shared-Weights Extender and Gradient Voting for Neural Network Expansion](https://mlsanity.com/paper/2509.18842/) (2025)

## Built on by (most cited first, 1 of 1)

- [When is Warmstarting Effective for Scaling Language Models?](https://mlsanity.com/paper/2605.13405/) (2026)

## Abstract (arXiv)

Width expansion offers a practical route to reuse smaller causal-language-model checkpoints, but selecting a widened warm start is not solved by zero-step preservation alone. We study dense width growth as a candidate-selection problem over full training states, including copied weights, optimizer moments, and scheduler state. In a small-scale TinyStories proxy, we compare exact-copy, perturbative, asymmetric-reset, and structured non-clone warm starts under matched continuation budgets. We evaluate zero-step preservation, short-lag probe metrics, and downstream continuation utility in deterministic and stochastic regimes. The picture is mixed and partially replicated through a reduced-pool seed-1 check. Exact-copy symmetric warm starts rank first in every completed 16-step probe and in the completed stochastic 128-step continuations at seed-0 steps 1000 and 2000 plus reduced seed-1 step 2000. By contrast, the structured non-clone challenger wins deterministic 128-step continuation. Early escape from the inherited cloned subspace is therefore not a universal selector: it helps in long deterministic continuation, but it misleads at short lag and under stochastic continuation. The result is narrow but useful: for dense width growth at this scale, preservation is not a universal ranking criterion, and the best replacement signal depends on both regime and lag budget.