mlsanityData to 14 Oct 2026

Preservation Is Not Enough for Width Growth: Regime-Sensitive Selection of Dense LM Warm Starts

arXiv 2604.04281 ↗First posted 5 Apr 2026cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Width expansion offers a practical route to reuse smaller causal-language-model checkpoints, but selecting a widened warm start is not solved by zero-step preservation alone. We study dense width growth as a candidate-selection problem over full training states, including copied weights, optimizer moments, and scheduler state. In a small-scale TinyStories proxy, we compare exact-copy, perturbative, asymmetric-reset, and structured non-clone warm starts under matched continuation budgets. We evaluate zero-step preservation, short-lag probe metrics, and downstream continuation utility in deterministic and stochastic regimes. The picture is mixed and partially replicated through a reduced-pool seed-1 check. Exact-copy symmetric warm starts rank first in every completed 16-step probe and in the completed stochastic 128-step continuations at seed-0 steps 1000 and 2000 plus reduced seed-1 step 2000. By contrast, the structured non-clone challenger wins deterministic 128-step continuation. Early escape from the inherited cloned subspace is therefore not a universal selector: it helps in long deterministic continuation, but it misleads at short lag and under stochastic continuation. The result is narrow but useful: for dense width growth at this scale, preservation is not a universal ranking criterion, and the best replacement signal depends on both regime and lag budget.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 16 of its 17 references that are in the map. First posted on arXiv 5 Apr 2026.

Builds on

16 of 17 references.

References · 16

  1. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 449
  2. 2015Net2Net: Accelerating Learning via Knowledge Transfercited by 250
  3. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 183
  4. 2023Learning to Grow Pretrained Models for Efficient Transformer Trainingcited by 83
  5. 2022Cramming: Training a Language Model on a Single GPU in One Daycited by 81
  6. 2021bert2BERT: Towards Reusable Pretrained Language Modelscited by 70
  7. 2016Network Morphismcited by 65
  8. 2022Staged Training for Transformer Language Modelscited by 49
  9. 2024Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Trainingcited by 43
  10. 2024Scaling Smart: Accelerating Large Language Model Pre-training with Small Model Initializationcited by 29
  11. 2023LEMON: Lossless model expansioncited by 23
  12. 2025Efficient Construction of Model Family through Progressive Training Using Model Expansioncited by 8
4 more
  1. 2024Landscape-Aware Growing: The Power of a Little LAGcited by 4
  2. 2024Warmstarting for Scaling Language Modelscited by 4
  3. 2026SPARKLING: Balancing Signal Preservation and Symmetry Breaking for Width-Progressive Learningcited by 4
  4. 2025Shared-Weights Extender and Gradient Voting for Neural Network Expansioncited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026When is Warmstarting Effective for Scaling Language Models?cited by 0