mlsanityData to 7 Oct 2026

On the Transformer Growth for Progressive BERT Training

arXiv 2010.12562 ↗First posted 23 Oct 2020cs.CL cs.LGcited by 49

Not on the map. Why?

Abstract (arXiv)

Due to the excessive cost of large-scale language model pre-training, considerable efforts have been made to train BERT progressively -- start from an inferior but low-cost model and gradually grow the model to increase the computational complexity. Our objective is to advance the understanding of Transformer growth and discover principles that guide progressive training. First, we find that similar to network architecture search, Transformer growth also favors compound scaling. Specifically, while existing methods only conduct network growth in a single dimension, we observe that it is beneficial to use compound growth operators and balance multiple dimensions (e.g., depth, width, and input length of the model). Moreover, we explore alternative growth operators in each dimension via controlled comparison to give operator selection practical guidance. In light of our analyses, the proposed method speeds up BERT pre-training by 73.6% and 82.2% for the base and large models respectively, while achieving comparable performances

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). First posted on arXiv 23 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 10; 2023: 11; 2024: 14; 2025: 9; 2026: 1.

49 papers.

Sort:

By area:

  1. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 207
  2. 2024LLaMA Pro: Progressive LLaMA with Block Expansioncited by 99
  3. 2022Cramming: Training a Language Model on a Single GPU in One Daycited by 81
  4. 2023Learning to Grow Pretrained Models for Efficient Transformer Trainingcited by 81
  5. 2021bert2BERT: Towards Reusable Pretrained Language Modelscited by 70
  6. 2022ELLE: Efficient Lifelong Pre-training for Emerging Datacited by 70
  7. 2024InsCL: A Data-efficient Continual Learning Paradigm for Fine-tuning Large Language Models with Instructionscited by 54
  8. 2022Staged Training for Transformer Language Modelscited by 49
  9. 2021Knowledge Inheritance for Pre-trained Language Modelscited by 38
  10. 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
  11. 2023A Survey on Efficient Training of Transformerscited by 37
  12. 2022Automated Progressive Learning for Efficient Training of Vision Transformerscited by 36
37 more
  1. 2023Masked Structural Growth for 2x Faster Language Model Pre-trainingcited by 32
  2. 2023Efficient Language Model Training through Cross-Lingual and Progressive Transfer Learningcited by 26
  3. 2023On Efficient Training of Large-Scale Deep Learning Models: A Literature Reviewcited by 26
  4. 2023FLM-101B: An Open LLM and How to Train It with 100K Budgetcited by 22
  5. 2022Transkimmer: Transformer Learns to Layer-wise Skimcited by 21
  6. 2023Seeking Neural Nuggets: Knowledge Transfer in Large Language Models from a Parametric Perspectivecited by 17
  7. 2023Contrastive Alignment of Vision to Language Through Parameter-Efficient Transfer Learningcited by 16
  8. 2023ConPET: Continual Parameter-Efficient Tuning for Large Language Modelscited by 16
  9. 2023Reusing Pretrained Models by Multi-linear Operators for Efficient Trainingcited by 15
  10. 2024A Little Help Goes a Long Way: Efficient LLM Training by Leveraging Small LMscited by 15
  11. 2021UCPhrase: Unsupervised Context-aware Quality Phrase Taggingcited by 14
  12. 2022Multi-CLS BERT: An Efficient Alternative to Traditional Ensemblingcited by 14
  13. 2022Layer Freezing & Data Sieving: Missing Pieces of a Generic Framework for Sparse Trainingcited by 11
  14. 2024Preparing Lessons for Progressive Training on Language Modelscited by 10
  15. 2024Efficient Stagewise Pretraining via Progressive Subnetworkscited by 10
  16. 2024DLO: Dynamic Layer Operation for Efficient Vertical Scaling of LLMscited by 9
  17. 2024Model-GLUE: Democratized LLM Scaling for A Large Model Zoo in the Wildcited by 9
  18. 2024Beyond Next Token Prediction: Patch-Level Training for Large Language Modelscited by 7
  19. 2024Towards Incremental Learning in Large Language Models: A Critical Reviewcited by 6
  20. 2024Stacking as Accelerated Gradient Descentcited by 5
  21. 2025LESA: Learnable LLM Layer Scaling-Upcited by 5
  22. 2021Training ELECTRA Augmented with Multi-word Selectioncited by 4
  23. 2023Deep Fusion: Efficient Network Training via Pre-trained Initializationscited by 4
  24. 2024Efficient Training of Large Vision Models via Advanced Automated Progressive Learningcited by 4
  25. 2025Curriculum-Guided Layer Scaling for Language Model Pretrainingcited by 4
  26. 2022FPT: Improving Prompt Tuning Efficiency via Progressive Trainingcited by 3
  27. 2025Flexible Realignment of Language Modelscited by 3
  28. 2024Early Period of Training Impacts Adaptation for Out-of-Distribution Generalization: An Empirical Studycited by 1
  29. 2025Scaling depth capacity via zero/one-layer model expansioncited by 1
  30. 2025Efficient Training for Human Video Generation with Entropy-Guided Prioritized Progressive Learningcited by 1
  31. 2024A Multi-Level Framework for Accelerating Training Transformer Modelscited by 0
  32. 2024Manifold Metric: A Loss Landscape Approach for Predicting Model Performancecited by 0
  33. 2025Adding Alignment Control to Language Modelscited by 0
  34. 2025A multilevel approach to accelerate the training of Transformerscited by 0
  35. 2025ScaleNet: Scaling up Pretrained Neural Networks with Incremental Parameterscited by 0
  36. 2025Efficient-Husformer: Efficient Multimodal Transformer Hyperparameter Optimization for Stress and Cognitive Loadscited by 0
  37. 2026Quantifying Depth Sufficiency in Residual Neural Networks: A First-Order Criterioncited by 0