# Masked Structural Growth for 2x Faster Language Model Pre-training

arXiv: https://arxiv.org/abs/2305.02869 · HTML: https://mlsanity.com/paper/2305.02869/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 43 references that are in the map.
- First posted on arXiv 4 May 2023.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [Green AI](https://mlsanity.com/paper/1907.10597/) (2019)
- [Net2Net: Accelerating Learning via Knowledge Transfer](https://mlsanity.com/paper/1511.05641/) (2015)
- [Structured Pruning of Large Language Models](https://mlsanity.com/paper/1910.04732/) (2019)
- [Structured Pruning Learns Compact and Accurate Models](https://mlsanity.com/paper/2204.00408/) (2022)
- [Learning to Grow Pretrained Models for Efficient Transformer Training](https://mlsanity.com/paper/2303.00980/) (2023)
- [EarlyBERT: Efficient BERT Training via Early-bird Lottery Tickets](https://mlsanity.com/paper/2101.00063/) (2020)
- [bert2BERT: Towards Reusable Pretrained Language Models](https://mlsanity.com/paper/2110.07143/) (2021)
- [ELLE: Efficient Lifelong Pre-training for Emerging Data](https://mlsanity.com/paper/2203.06311/) (2022)
- [GradMax: Growing Neural Networks using Gradient Information](https://mlsanity.com/paper/2201.05125/) (2022)
- [Network Morphism](https://mlsanity.com/paper/1603.01670/) (2016)
- [On the Transformer Growth for Progressive BERT Training](https://mlsanity.com/paper/2010.12562/) (2020)
- [Staged Training for Transformer Language Models](https://mlsanity.com/paper/2203.06211/) (2022)
- [Knowledge Inheritance for Pre-trained Language Models](https://mlsanity.com/paper/2105.13880/) (2021)
- [Compute-Efficient Deep Learning: Algorithmic Trends and Opportunities](https://mlsanity.com/paper/2210.06640/) (2022)
- [Automated Progressive Learning for Efficient Training of Vision Transformers](https://mlsanity.com/paper/2203.14509/) (2022)
- [Progressively Stacking 2.0: A Multi-stage Layerwise Training Method for BERT Training Speedup](https://mlsanity.com/paper/2011.13635/) (2020)
- [AutoGrow: Automatic Layer Growing in Deep Convolutional Networks](https://mlsanity.com/paper/1906.02909/) (2019)

## Built on by (most cited first, 30 of 33)

- [Efficient Large Language Models: A Survey](https://mlsanity.com/paper/2312.03863/) (2023)
- [Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Training](https://mlsanity.com/paper/2405.15319/) (2024)
- [FLM-101B: An Open LLM and How to Train It with 100K Budget](https://mlsanity.com/paper/2309.03852/) (2023)
- [On the Inductive Bias of Stacking Towards Improving Reasoning](https://mlsanity.com/paper/2409.19044/) (2024)
- [Composable Function-preserving Expansions for Transformer Architectures](https://mlsanity.com/paper/2308.06103/) (2023)
- [52B to 1T: Lessons Learned via Tele-FLM Series](https://mlsanity.com/paper/2407.02783/) (2024)
- [Efficient Stagewise Pretraining via Progressive Subnetworks](https://mlsanity.com/paper/2402.05913/) (2024)
- [DLO: Dynamic Layer Operation for Efficient Vertical Scaling of LLMs](https://mlsanity.com/paper/2407.11030/) (2024)
- [Beyond Next Token Prediction: Patch-Level Training for Large Language Models](https://mlsanity.com/paper/2407.12665/) (2024)
- [From Growing to Looping: A Unified View of Iterative Computation in LLMs](https://mlsanity.com/paper/2602.16490/) (2026)
- [Do Depth-Grown Models Overcome the Curse of Depth? An In-Depth Analysis](https://mlsanity.com/paper/2512.08819/) (2025)
- [LESA: Learnable LLM Layer Scaling-Up](https://mlsanity.com/paper/2502.13794/) (2025)
- [Landscape-Aware Growing: The Power of a Little LAG](https://mlsanity.com/paper/2406.02469/) (2024)
- [Curriculum-Guided Layer Scaling for Language Model Pretraining](https://mlsanity.com/paper/2506.11389/) (2025)
- [Optimal Transport Depth Up-Scaling](https://mlsanity.com/paper/2508.08011/) (2025)
- [SPARKLING: Balancing Signal Preservation and Symmetry Breaking for Width-Progressive Learning](https://mlsanity.com/paper/2602.02472/) (2026)
- [The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws](https://mlsanity.com/paper/2501.12486/) (2025)
- [STEP: Staged Parameter-Efficient Pre-training for Large Language Models](https://mlsanity.com/paper/2504.04151/) (2025)
- [Frozen Layers: Memory-efficient Many-fidelity Hyperparameter Optimization](https://mlsanity.com/paper/2504.10735/) (2025)
- [Expanding Foundational Language Capabilities in Open-Source LLMs through a Korean Case Study](https://mlsanity.com/paper/2509.03972/) (2025)
- [Reusing Overtrained Language Models Saturates Scaling](https://mlsanity.com/paper/2510.06548/) (2025)
- [Matrix-Driven Identification and Reconstruction of LLM Weight Homology](https://mlsanity.com/paper/2508.06309/) (2025)
- [Beyond Sunk Costs: Boosting LLM Pre-training Efficiency via Orthogonal Growth of Mixture-of-Experts](https://mlsanity.com/paper/2510.08008/) (2025)
- [FrugalPrompt: Reducing Contextual Overhead in Large Language Models via Token Attribution](https://mlsanity.com/paper/2510.16439/) (2025)
- [Scaling depth capacity via zero/one-layer model expansion](https://mlsanity.com/paper/2511.04981/) (2025)
- [A Mean Field Ansatz for Zero-Shot Weight Transfer](https://mlsanity.com/paper/2408.08681/) (2024)
- [Long-Tailed Data Classification by Increasing and Decreasing Neurons During Training](https://mlsanity.com/paper/2507.09940/) (2025)
- [SCALE: Upscaled Continual Learning of Large Language Models](https://mlsanity.com/paper/2511.03270/) (2025)
- [Nexusformer: Nonlinear Attention Expansion for Stable and Inheritable Transformer Scaling](https://mlsanity.com/paper/2604.19147/) (2026)
- [When is Warmstarting Effective for Scaling Language Models?](https://mlsanity.com/paper/2605.13405/) (2026)

## Abstract (arXiv)

Accelerating large language model pre-training is a critical issue in present research. In this paper, we focus on speeding up pre-training by progressively growing from a small Transformer structure to a large one. There are two main research problems associated with progressive growth: determining the optimal growth schedule, and designing efficient growth operators. In terms of growth schedule, the impact of each single dimension on a schedule's efficiency is under-explored by existing work. Regarding the growth operators, existing methods rely on the initialization of new weights to inherit knowledge, and achieve only non-strict function preservation, limiting further improvements on training dynamics. To address these issues, we propose Masked Structural Growth (MSG), including (i) growth schedules involving all possible dimensions and (ii) strictly function-preserving growth operators that is independent of the initialization of new weights. Experiments show that MSG is significantly faster than related work: we achieve up to 2.2x speedup in pre-training different types of language models while maintaining comparable or better downstream performances. Code is publicly available at https://github.com/cofe-ai/MSG.