mlsanityData to 7 Oct 2026

Masked Structural Growth for 2x Faster Language Model Pre-training

arXiv 2305.02869 ↗First posted 4 May 2023cs.CLcited by 32

Not on the map. Why?

Abstract (arXiv)

Accelerating large language model pre-training is a critical issue in present research. In this paper, we focus on speeding up pre-training by progressively growing from a small Transformer structure to a large one. There are two main research problems associated with progressive growth: determining the optimal growth schedule, and designing efficient growth operators. In terms of growth schedule, the impact of each single dimension on a schedule's efficiency is under-explored by existing work. Regarding the growth operators, existing methods rely on the initialization of new weights to inherit knowledge, and achieve only non-strict function preservation, limiting further improvements on training dynamics. To address these issues, we propose Masked Structural Growth (MSG), including (i) growth schedules involving all possible dimensions and (ii) strictly function-preserving growth operators that is independent of the initialization of new weights. Experiments show that MSG is significantly faster than related work: we achieve up to 2.2x speedup in pre-training different types of language models while maintaining comparable or better downstream performances. Code is publicly available at https://github.com/cofe-ai/MSG.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 30 of its 43 references that are in the map. First posted on arXiv 4 May 2023.

Builds on

30 of 43 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2016Layer Normalizationcited by 5,579
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  9. 2016Pointer Sentinel Mixture Modelscited by 2,992
  10. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  11. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  12. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
18 more
  1. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  2. 2019Green AIcited by 620
  3. 2015Net2Net: Accelerating Learning via Knowledge Transfercited by 245
  4. 2019Structured Pruning of Large Language Modelscited by 201
  5. 2022Structured Pruning Learns Compact and Accurate Modelscited by 167
  6. 2023Learning to Grow Pretrained Models for Efficient Transformer Trainingcited by 81
  7. 2020EarlyBERT: Efficient BERT Training via Early-bird Lottery Ticketscited by 79
  8. 2021bert2BERT: Towards Reusable Pretrained Language Modelscited by 70
  9. 2022ELLE: Efficient Lifelong Pre-training for Emerging Datacited by 70
  10. 2022GradMax: Growing Neural Networks using Gradient Informationcited by 64
  11. 2016Network Morphismcited by 63
  12. 2020On the Transformer Growth for Progressive BERT Trainingcited by 49
  13. 2022Staged Training for Transformer Language Modelscited by 49
  14. 2021Knowledge Inheritance for Pre-trained Language Modelscited by 38
  15. 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
  16. 2022Automated Progressive Learning for Efficient Training of Vision Transformerscited by 36
  17. 2020Progressively Stacking 2.0: A Multi-stage Layerwise Training Method for BERT Training Speedupcited by 31
  18. 2019AutoGrow: Automatic Layer Growing in Deep Convolutional Networkscited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 8; 2025: 15; 2026: 6.

32 papers.

Sort:

By area:

  1. 2023Efficient Large Language Models: A Surveycited by 183
  2. 2024Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Trainingcited by 42
  3. 2023FLM-101B: An Open LLM and How to Train It with 100K Budgetcited by 22
  4. 2024On the Inductive Bias of Stacking Towards Improving Reasoningcited by 18
  5. 2023Composable Function-preserving Expansions for Transformer Architecturescited by 11
  6. 202452B to 1T: Lessons Learned via Tele-FLM Seriescited by 11
  7. 2024Efficient Stagewise Pretraining via Progressive Subnetworkscited by 10
  8. 2024DLO: Dynamic Layer Operation for Efficient Vertical Scaling of LLMscited by 9
  9. 2024Beyond Next Token Prediction: Patch-Level Training for Large Language Modelscited by 7
  10. 2026From Growing to Looping: A Unified View of Iterative Computation in LLMscited by 6
  11. 2025LESA: Learnable LLM Layer Scaling-Upcited by 5
  12. 2025Do Depth-Grown Models Overcome the Curse of Depth? An In-Depth Analysiscited by 5
20 more
  1. 2024Landscape-Aware Growing: The Power of a Little LAGcited by 4
  2. 2025Curriculum-Guided Layer Scaling for Language Model Pretrainingcited by 4
  3. 2025Optimal Transport Depth Up-Scalingcited by 4
  4. 2026SPARKLING: Balancing Signal Preservation and Symmetry Breaking for Width-Progressive Learningcited by 4
  5. 2025The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Lawscited by 2
  6. 2025STEP: Staged Parameter-Efficient Pre-training for Large Language Modelscited by 2
  7. 2025Frozen Layers: Memory-efficient Many-fidelity Hyperparameter Optimizationcited by 2
  8. 2025Expanding Foundational Language Capabilities in Open-Source LLMs through a Korean Case Studycited by 2
  9. 2025Reusing Overtrained Language Models Saturates Scalingcited by 2
  10. 2025Matrix-Driven Identification and Reconstruction of LLM Weight Homologycited by 1
  11. 2025Beyond Sunk Costs: Boosting LLM Pre-training Efficiency via Orthogonal Growth of Mixture-of-Expertscited by 1
  12. 2025FrugalPrompt: Reducing Contextual Overhead in Large Language Models via Token Attributioncited by 1
  13. 2025Scaling depth capacity via zero/one-layer model expansioncited by 1
  14. 2024A Mean Field Ansatz for Zero-Shot Weight Transfercited by 0
  15. 2025Long-Tailed Data Classification by Increasing and Decreasing Neurons During Trainingcited by 0
  16. 2025SCALE: Upscaled Continual Learning of Large Language Modelscited by 0
  17. 2026Nexusformer: Nonlinear Attention Expansion for Stable and Inheritable Transformer Scalingcited by 0
  18. 2026When is Warmstarting Effective for Scaling Language Models?cited by 0
  19. 2026What Does Model Growth Really Add? Functional Capacity Beyond Parameter Countcited by 0
  20. 2026Mixture of Training: Recombining Small-Scale Scaffolded Pretraining Runs into a Larger Language Modelcited by 0