Understanding Parameter Sharing in Transformers
Not on the map. Why?
Abstract (arXiv)
Parameter sharing has proven to be a parameter-efficient approach. Previous work on Transformers has focused on sharing parameters in different layers, which can improve the performance of models with limited parameters by increasing model depth. In this paper, we study why this approach works from two perspectives. First, increasing model depth makes the model more complex, and we hypothesize that the reason is related to model complexity (referring to FLOPs). Secondly, since each shared parameter will participate in the network computation several times in forward propagation, its corresponding gradient will have a different range of values from the original model, which will affect the model convergence. Based on this, we hypothesize that training convergence may also be one of the reasons. Through further analysis, we show that the success of this approach can be largely attributed to better convergence, with only a small part due to the increased model complexity. Inspired by this, we tune the training hyperparameters related to model convergence in a targeted manner. Experiments on 8 machine translation tasks show that our model achieves competitive performance with only half the model complexity of parameter sharing models.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 38 of its 46 references that are in the map. First posted on arXiv 15 Jun 2023.
Builds on
38 of 46 references.
References · 38
- 2017Attention Is All You Needcited by 60,361
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2015Distilling the Knowledge in a Neural Networkcited by 10,324
- 2016Layer Normalizationcited by 5,669
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
- 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
- 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
- 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
- 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
- 2017Mixed Precision Trainingcited by 1,156
- 2018Self-Attention with Relative Position Representationscited by 1,120
26 more
- 2016Sequence-Level Knowledge Distillationcited by 801
- 2018Universal Transformerscited by 647
- 2019Learning Deep Transformer Models for Machine Translationcited by 309
- 2018Adaptive Input Representations for Neural Language Modelingcited by 226
- 2020Understanding the Difficulty of Training Transformerscited by 210
- 2018Scaling Neural Machine Translationcited by 204
- 2019The Evolved Transformercited by 180
- 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 178
- 2017Soft Weight-Sharing for Neural Network Compressioncited by 99
- 2015Representation Benefits of Deep Feedforward Networkscited by 86
- 2021Lessons on Parameter Sharing across Layers in Transformerscited by 85
- 2020Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformerscited by 67
- 2020Very Deep Transformers for Neural Machine Translationcited by 50
- 2017DiracNets: Training Very Deep Neural Networks Without Skip-Connectionscited by 47
- 2017Ensemble Distillation for Neural Machine Translationcited by 39
- 2023Language Models are Few-shot Learners for Prognostic Predictioncited by 37
- 2020Towards Fully 8-bit Integer Inference for the Transformer Modelcited by 35
- 2019Modeling Recurrence for Transformercited by 26
- 2020Weight Distillation: Transferring the Knowledge in Neural Network Parameterscited by 23
- 2019MUSE: Parallel Multi-Scale Attention for Sequence to Sequence Learningcited by 19
- 2017Weighted Transformer Network for Machine Translationcited by 16
- 2019Depth Growing for Neural Machine Translationcited by 13
- 2020Multi-branch Attentive Transformercited by 10
- 2018Pieces of Eight: 8-bit Neural Machine Translationcited by 7
- 2021Recurrent multiple shared layers in Depth for Neural Machine Translationcited by 3
- 2021Recurrent Stacking of Layers in Neural Networks: An Application to Neural Machine Translationcited by 1
Built on by
Citing papers per year: 2024: 2; 2025: 1.
3 papers.