mlsanityData to 14 Oct 2026

Speeding up Deep Model Training by Sharing Weights and Then Unsharing

arXiv 2110.03848 ↗First posted 8 Oct 2021cs.LG cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

We propose a simple and efficient approach for training the BERT model. Our approach exploits the special structure of BERT that contains a stack of repeated modules (i.e., transformer encoders). Our proposed approach first trains BERT with the weights shared across all the repeated modules till some point. This is for learning the commonly shared component of weights across all repeated layers. We then stop weight sharing and continue training until convergence. We present theoretic insights for training by sharing weights then unsharing with analysis for simplified models. Empirical experiments on the BERT model show that our method yields better performance of trained models, and significantly reduces the number of training iterations.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 30 of its 39 references that are in the map. First posted on arXiv 8 Oct 2021.

Builds on

30 of 39 references.

References · 30

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  8. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  9. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  10. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  11. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  12. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
18 more
  1. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  2. 2020On Layer Normalization in the Transformer Architecturecited by 922
  3. 2015Accurate Image Super-Resolution Using Very Deep Convolutional Networkscited by 903
  4. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 834
  5. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  6. 2019Deep Equilibrium Modelscited by 615
  7. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 543
  8. 2015Deep Speech 2: End-to-End Speech Recognition in English and Mandarincited by 451
  9. 2019Learning Deep Transformer Models for Machine Translationcited by 309
  10. 2015Net2Net: Accelerating Learning via Knowledge Transfercited by 250
  11. 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 184
  12. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 169
  13. 2017Multi-level Residual Networks from Dynamical Systems Viewcited by 59
  14. 2018Trellis Networks for Sequence Modelingcited by 52
  15. 2018Recurrent Stacking of Layers for Compact Neural Machine Translation Modelscited by 38
  16. 2020Deeper Insights into Weight Sharing in Neural Architecture Searchcited by 24
  17. 2019Global Convergence of Gradient Descent for Deep Linear Residual Networkscited by 16
  18. 2020Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networkscited by 6

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 0; 2024: 3; 2025: 2.

7 papers.

  1. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 209
  2. 2022ERNIE-Search: Bridging Cross-Encoder with Dual-Encoder via Self On-the-fly Distillation for Dense Passage Retrievalcited by 41
  3. 2025Parallel Scaling Law for Language Modelscited by 30
  4. 2024Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Taskscited by 22
  5. 2024Preparing Lessons for Progressive Training on Language Modelscited by 10
  6. 2024A Multi-Level Framework for Accelerating Training Transformer Modelscited by 0
  7. 2025A multilevel approach to accelerate the training of Transformerscited by 0