mlsanityData to 7 Oct 2026

Lessons on Parameter Sharing across Layers in Transformers

arXiv 2104.06022 ↗First posted 13 Apr 2021cs.CL cs.LGcited by 85

Not on the map. Why?

Abstract (arXiv)

We propose a parameter sharing method for Transformers (Vaswani et al., 2017). The proposed approach relaxes a widely used technique, which shares parameters for one layer with all layers such as Universal Transformers (Dehghani et al., 2019), to increase the efficiency in the computational time. We propose three strategies: Sequence, Cycle, and Cycle (rev) to assign parameters to each layer. Experimental results show that the proposed strategies are efficient in the parameter size and computational time. Moreover, we indicate that the proposed strategies are also effective in the configuration where we use many training data such as the recent WMT competition.

Key facts. Cited by 85 papers in the map (data through 7 Oct 2026). Builds on 30 of its 38 references that are in the map. First posted on arXiv 13 Apr 2021.

Builds on

30 of 38 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  6. 2016Pointer Sentinel Mixture Modelscited by 2,992
  7. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  8. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  9. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  10. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  11. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  12. 2020On Layer Normalization in the Transformer Architecturecited by 906
18 more
  1. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 784
  2. 2018Universal Transformerscited by 613
  3. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 479
  4. 2019Learning Deep Transformer Models for Machine Translationcited by 309
  5. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  6. 2020Understanding the Difficulty of Training Transformerscited by 210
  7. 2017Regularizing and Optimizing LSTM Language Modelscited by 205
  8. 2018Scaling Neural Machine Translationcited by 204
  9. 2020fairseq S2T: Fast Speech-to-Text Modeling with fairseqcited by 174
  10. 2019Transformers without Tears: Improving the Normalization of Self-Attentioncited by 153
  11. 2017On the State of the Art of Evaluation in Neural Language Modelscited by 103
  12. 2019Improving Transformer Models by Reordering their Sublayerscited by 52
  13. 2018Recurrent Stacking of Layers for Compact Neural Machine Translation Modelscited by 38
  14. 2020Shallow-to-Deep Training for Neural Machine Translationcited by 36
  15. 2019Sharing Attention Weights for Fast Transformercited by 34
  16. 2021Rethinking Perturbations in Encoder-Decoders for Fast Trainingcited by 17
  17. 2018Direct Output Connection for a High-Rank Language Modelcited by 10
  18. 2020All Word Embeddings from One Embeddingcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 6; 2023: 15; 2024: 20; 2025: 21; 2026: 19.

85 papers.

Sort:

By area:

  1. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 372
  2. 2025Scaling Latent Reasoning via Looped Language Modelscited by 145
  3. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 120
  4. 2024Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRAcited by 72
  5. 2025Inner Thinking Transformer: Leveraging Dynamic Depth Scaling to Foster Adaptive Internal Thinkingcited by 56
  6. 2024MoEUT: Mixture-of-Experts Universal Transformerscited by 55
  7. 2021A Survey on Green Deep Learningcited by 52
  8. 2023Spike No More: Stabilizing the Pre-training of Large Language Modelscited by 51
  9. 2026LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulationcited by 48
  10. 2023SPEED: Speculative Pipelined Execution for Efficient Decodingcited by 47
  11. 2022A Survey on Model Compression and Acceleration for Pretrained Language Modelscited by 45
  12. 2025Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrencecited by 41
38 more
  1. 2024KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharingcited by 38
  2. 2021On Compositional Generalization of Neural Machine Translationcited by 34
  3. 2024Basis Sharing: Cross-Layer Parameter Sharing for Large Language Model Compressioncited by 34
  4. 2023Sparse Universal Transformercited by 28
  5. 2024On Expressive Power of Looped Transformers: Theoretical Analysis and Enhancement via Timestep Encodingcited by 28
  6. 2023CoTFormer: A Chain-of-Thought Driven Architecture with Budget-Adaptive Computation Cost at Inferencecited by 27
  7. 2022EdgeFormer: A Parameter-Efficient Transformer for On-Device Seq2seq Generationcited by 21
  8. 2021Interpreting Deep Learning Models in Natural Language Processing: A Reviewcited by 19
  9. 2024PRoLoRA: Partial Rotation Empowers More Parameter-Efficient LoRAcited by 16
  10. 2023One Wide Feedforward is All You Needcited by 15
  11. 2024A Comprehensive Survey of Compression Algorithms for Language Modelscited by 15
  12. 2023Towards energy-efficient Deep Learning: An overview of energy-efficient approaches along the Deep Learning Lifecyclecited by 13
  13. 2024Adapter-X: A Novel General Parameter-Efficient Fine-Tuning Framework for Visioncited by 13
  14. 2024TroL: Traversal of Layers for Large Language and Vision Modelscited by 13
  15. 2026Training-Free Looped Transformerscited by 12
  16. 2026Solve the Loop: Attractor Models for Language and Reasoningcited by 11
  17. 2024Dynamic Layer Tying for Parameter-Efficient Transformerscited by 10
  18. 2022Spiking Neural Networks for event-based action recognition: A new task to understand their advantagecited by 9
  19. 2024Exploring Learngene via Stage-wise Weight Sharing for Initializing Variable-sized Modelscited by 9
  20. 2024BlockLLM: Memory-Efficient Adaptation of LLMs by Selecting and Optimizing the Right Coordinate Blockscited by 9
  21. 2026SpiralFormer: Looped Transformers Can Learn Hierarchical Dependencies via Multi-Resolution Recursioncited by 9
  22. 2022Streaming parallel transducer beam search with fast-slow cascaded encoderscited by 8
  23. 2025MeSH: Memory-as-State-Highways for Recursive Transformerscited by 8
  24. 2023On the Opportunities of Green Computing: A Surveycited by 7
  25. 2024MoS: Unleashing Parameter Efficiency of Low-Rank Adaptation with Mixture of Shardscited by 7
  26. 2025Chain-of-Experts: Unlocking the Communication Power of Mixture-of-Experts Modelscited by 7
  27. 2024Increasing Model Capacity for Free: A Simple Strategy for Parameter Efficient Fine-tuningcited by 6
  28. 2025RingFormer: Rethinking Recurrent Transformer with Adaptive Level Signalscited by 6
  29. 2025Self-Supervised Transformers as Iterative Solution Improvers for Constraint Satisfactioncited by 6
  30. 2025Layer-wise dynamic rank for compressing large language modelscited by 6
  31. 2026From Growing to Looping: A Unified View of Iterative Computation in LLMscited by 6
  32. 2025Zero Token-Driven Deep Thinking in LLMs: Unlocking the Full Potential of Existing Parameters via Cyclic Refinementcited by 5
  33. 2025Share Your Attention: Transformer Weight Sharing via Matrix-based Dictionary Learningcited by 5
  34. 2026LT2: Linear-Time Looped Transformerscited by 5
  35. 2024Head-wise Shareable Attention for Large Language Modelscited by 4
  36. 2025FlexiGPT: Pruning and Extending Large Language Models with Low-Rank Weight Sharingcited by 4
  37. 2026Looping Back to Move Forward: Recursive Transformers for Efficient and Flexible Large Multimodal Modelscited by 4
  38. 2026Loop the Loopies!cited by 4

50 most cited of 85 All 85 (JSON)