# Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Training

arXiv: https://arxiv.org/abs/2405.15319 · HTML: https://mlsanity.com/paper/2405.15319/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 47 references that are in the map.
- First posted on arXiv 24 May 2024.

## Builds on (most cited first, 30 of 32)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Crowdsourcing Multiple Choice Science Questions](https://mlsanity.com/paper/1707.06209/) (2017)
- [OLMo: Accelerating the Science of Language Models](https://mlsanity.com/paper/2402.00838/) (2024)
- [Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research](https://mlsanity.com/paper/2402.00159/) (2024)
- [Net2Net: Accelerating Learning via Knowledge Transfer](https://mlsanity.com/paper/1511.05641/) (2015)
- [Retrieve, Program, Repeat: Complex Knowledge Base Question Answering via Alternate Meta-learning](https://mlsanity.com/paper/2010.15875/) (2020)
- [SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling](https://mlsanity.com/paper/2312.15166/) (2023)
- [LLaMA Pro: Progressive LLaMA with Block Expansion](https://mlsanity.com/paper/2401.02415/) (2024)
- [Understanding Emergent Abilities of Language Models from the Loss Perspective](https://mlsanity.com/paper/2403.15796/) (2024)
- [Learning to Grow Pretrained Models for Efficient Transformer Training](https://mlsanity.com/paper/2303.00980/) (2023)
- [bert2BERT: Towards Reusable Pretrained Language Models](https://mlsanity.com/paper/2110.07143/) (2021)
- [GradMax: Growing Neural Networks using Gradient Information](https://mlsanity.com/paper/2201.05125/) (2022)
- [Firefly Neural Architecture Descent: a General Approach for Growing Neural Networks](https://mlsanity.com/paper/2102.08574/) (2021)
- [Staged Training for Transformer Language Models](https://mlsanity.com/paper/2203.06211/) (2022)
- [No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Models](https://mlsanity.com/paper/2307.06440/) (2023)
- [Masked Structural Growth for 2x Faster Language Model Pre-training](https://mlsanity.com/paper/2305.02869/) (2023)
- [Progressively Stacking 2.0: A Multi-stage Layerwise Training Method for BERT Training Speedup](https://mlsanity.com/paper/2011.13635/) (2020)
- [LEMON: Lossless model expansion](https://mlsanity.com/paper/2310.07999/) (2023)
- [FLM-101B: An Open LLM and How to Train It with 100K Budget](https://mlsanity.com/paper/2309.03852/) (2023)

## Built on by (most cited first, 30 of 43)

- [LongCat-Flash Technical Report](https://mlsanity.com/paper/2509.01322/) (2025)
- [Scaling Smart: Accelerating Large Language Model Pre-training with Small Model Initialization](https://mlsanity.com/paper/2409.12903/) (2024)
- [Scaling Laws for Upcycling Mixture-of-Experts Language Models](https://mlsanity.com/paper/2502.03009/) (2025)
- [DLO: Dynamic Layer Operation for Efficient Vertical Scaling of LLMs](https://mlsanity.com/paper/2407.11030/) (2024)
- [Efficient Construction of Model Family through Progressive Training Using Model Expansion](https://mlsanity.com/paper/2504.00623/) (2025)
- [From Growing to Looping: A Unified View of Iterative Computation in LLMs](https://mlsanity.com/paper/2602.16490/) (2026)
- [Do Depth-Grown Models Overcome the Curse of Depth? An In-Depth Analysis](https://mlsanity.com/paper/2512.08819/) (2025)
- [AI Governance through Markets](https://mlsanity.com/paper/2501.17755/) (2025)
- [LESA: Learnable LLM Layer Scaling-Up](https://mlsanity.com/paper/2502.13794/) (2025)
- [Towards Alignment-Centric Paradigm: A Survey of Instruction Tuning in Large Language Models](https://mlsanity.com/paper/2508.17184/) (2025)
- [Curriculum-Guided Layer Scaling for Language Model Pretraining](https://mlsanity.com/paper/2506.11389/) (2025)
- [Optimal Transport Depth Up-Scaling](https://mlsanity.com/paper/2508.08011/) (2025)
- [SPARKLING: Balancing Signal Preservation and Symmetry Breaking for Width-Progressive Learning](https://mlsanity.com/paper/2602.02472/) (2026)
- [When Does Sparsity Mitigate the Curse of Depth in LLMs](https://mlsanity.com/paper/2603.15389/) (2026)
- [XRAG: eXamining the Core -- Benchmarking Foundational Components in Advanced Retrieval-Augmented Generation](https://mlsanity.com/paper/2412.15529/) (2024)
- [Large Language models for Time Series Analysis: Techniques, Applications, and Challenges](https://mlsanity.com/paper/2506.11040/) (2025)
- [A Layered Intuition -- Method Model with Scope Extension for LLM Reasoning](https://mlsanity.com/paper/2510.10592/) (2025)
- [Grow Up and Merge: Scaling Strategies for Efficient Language Adaptation](https://mlsanity.com/paper/2512.10772/) (2025)
- [Frozen Layers: Memory-efficient Many-fidelity Hyperparameter Optimization](https://mlsanity.com/paper/2504.10735/) (2025)
- [Mitigating Catastrophic Forgetting in Continual Learning through Model Growth](https://mlsanity.com/paper/2509.01213/) (2025)
- [Crown, Frame, Reverse: Layer-Wise Scaling Variants for LLM Pre-Training](https://mlsanity.com/paper/2509.06518/) (2025)
- [Reusing Overtrained Language Models Saturates Scaling](https://mlsanity.com/paper/2510.06548/) (2025)
- [How Model Growth, Recursion, and Boundary Operators Influence Scaling Exponents](https://mlsanity.com/paper/2609.19107/) (2026)
- [Model Steering: Learning with a Reference Model Improves Generalization Bounds and Scaling Laws](https://mlsanity.com/paper/2505.06699/) (2025)
- [All is Not Lost: LLM Recovery without Checkpoints](https://mlsanity.com/paper/2506.15461/) (2025)
- [Growing Transformers: Modular Composition and Layer-wise Expansion on a Frozen Substrate](https://mlsanity.com/paper/2507.07129/) (2025)
- [Beyond Sunk Costs: Boosting LLM Pre-training Efficiency via Orthogonal Growth of Mixture-of-Experts](https://mlsanity.com/paper/2510.08008/) (2025)
- [Scaling depth capacity via zero/one-layer model expansion](https://mlsanity.com/paper/2511.04981/) (2025)
- [Late-to-Early Training: LET LLMs Learn Earlier, So Faster and Better](https://mlsanity.com/paper/2602.05393/) (2026)
- [Preservation Is Not Enough for Width Growth: Regime-Sensitive Selection of Dense LM Warm Starts](https://mlsanity.com/paper/2604.04281/) (2026)

## Abstract (arXiv)

LLMs are computationally expensive to pre-train due to their large scale. Model growth emerges as a promising approach by leveraging smaller models to accelerate the training of larger ones. However, the viability of these model growth methods in efficient LLM pre-training remains underexplored. This work identifies three critical underlinetextitObstacles: (textitO1) lack of comprehensive evaluation, (textitO2) untested viability for scaling, and (textitO3) lack of empirical guidelines. To tackle textitO1, we summarize existing approaches into four atomic growth operators and systematically evaluate them in a standardized LLM pre-training setting. Our findings reveal that a depthwise stacking operator, called Gstack, exhibits remarkable acceleration in training, leading to decreased loss and improved overall performance on eight standard NLP benchmarks compared to strong baselines. Motivated by these promising results, we conduct extensive experiments to delve deeper into Gstack to address textitO2 and textitO3. For textitO2 (untested scalability), our study shows that Gstack is scalable and consistently performs well, with experiments up to 7B LLMs after growth and pre-training LLMs with 750B tokens. For example, compared to a conventionally trained 7B model using 300B tokens, our Gstack model converges to the same loss with 194B tokens, resulting in a 54.6% speedup. We further address textitO3 (lack of empirical guidelines) by formalizing guidelines to determine growth timing and growth factor for Gstack, making it practical in general LLM pre-training. We also provide in-depth discussions and comprehensive ablation studies of Gstack. Our code and pre-trained model are available at https://llm-stacking.github.io.