mlsanityData to 13 Oct 2026

Subformer: Exploring Weight Sharing for Parameter Efficiency in Generative Transformers

arXiv 2101.00234 ↗First posted 1 Jan 2021cs.CL cs.LGcited by 42

Not on the map. Why?

Abstract (arXiv)

Transformers have shown improved performance when compared to previous architectures for sequence processing such as RNNs. Despite their sizeable performance gains, as recently suggested, the model is computationally expensive to train and with a high parameter budget. In light of this, we explore parameter-sharing methods in Transformers with a specific focus on generative models. We perform an analysis of different parameter sharing/reduction methods and develop the Subformer. Our model combines sandwich-style parameter sharing, which overcomes naive cross-layer parameter sharing in generative models, and self-attentive embedding factorization (SAFE). Experiments on machine translation, abstractive summarization and language modeling show that the Subformer can outperform the Transformer even when using significantly fewer parameters.

Key facts. Cited by 42 papers in the map (data through 13 Oct 2026). Builds on 44 of its 58 references that are in the map. First posted on arXiv 1 Jan 2021.

Builds on

44 of 58 references.

References · 44

  1. 2017Attention Is All You Needcited by 60,106
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  4. 2014Adam: A Method for Stochastic Optimizationcited by 37,093
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,223
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,322
  7. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,884
  8. 2016Layer Normalizationcited by 5,624
  9. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,539
  10. 2016Pointer Sentinel Mixture Modelscited by 3,047
  11. 2014Sequence to Sequence Learning with Neural Networkscited by 2,981
  12. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,910
32 more
  1. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,779
  2. 2018Deep contextualized word representationscited by 2,286
  3. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,835
  4. 2020Reformer: The Efficient Transformercited by 1,662
  5. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,487
  6. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,288
  7. 2017Mixed Precision Trainingcited by 1,147
  8. 2015Teaching Machines to Read and Comprehendcited by 1,045
  9. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 810
  10. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 809
  11. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 797
  12. 2018Universal Transformerscited by 632
  13. 2019Deep Equilibrium Modelscited by 606
  14. 2019Text Summarization with Pretrained Encoderscited by 554
  15. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 436
  16. 2019Q8BERT: Quantized 8Bit BERTcited by 299
  17. 2016Using the Output Embedding to Improve Language Modelscited by 269
  18. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  19. 2019Pay Less Attention with Lightweight and Dynamic Convolutionscited by 211
  20. 2018Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinementcited by 195
  21. 2019Leveraging Pre-trained Checkpoints for Sequence Generation Taskscited by 182
  22. 2019The Evolved Transformercited by 180
  23. 2020Lite Transformer with Long-Short Range Attentioncited by 127
  24. 2017Controllable Abstractive Summarizationcited by 110
  25. 2018The Best of Both Worlds: Combining Recent Advances in Neural Machine Translationcited by 102
  26. 2020Compressing Large-Scale Transformer-Based Models: A Case Study on BERTcited by 95
  27. 2017Dynamic Evaluation of Neural Sequence Modelscited by 55
  28. 2019Pre-trained Language Model Representations for Language Generationcited by 34
  29. 2018An Analysis of Neural Language Modeling at Multiple Scalescited by 31
  30. 2018Exploiting Deep Representations for Neural Machine Translationcited by 23
  31. 2020DeLighT: Deep and Light-weight Transformercited by 17
  32. 2019DeFINE: DEep Factorized INput Token Embeddings for Neural Sequence Modelingcited by 10

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2022: 4; 2023: 8; 2024: 16; 2025: 7; 2026: 7.

42 papers.

Sort:

By area:

  1. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 217
  2. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  3. 2024Layer-Condensed KV Cache for Efficient Inference of Large Language Modelscited by 59
  4. 2023SPEED: Speculative Pipelined Execution for Efficient Decodingcited by 47
  5. 2022A Survey on Model Compression and Acceleration for Pretrained Language Modelscited by 45
  6. 2024KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharingcited by 39
  7. 2024Basis Sharing: Cross-Layer Parameter Sharing for Large Language Model Compressioncited by 35
  8. 2023The Efficiency Spectrum of Large Language Models: An Algorithmic Surveycited by 26
  9. 2022EdgeFormer: A Parameter-Efficient Transformer for On-Device Seq2seq Generationcited by 21
  10. 2024QSpec: Speculative Decoding with Complementary Quantization Schemescited by 18
  11. 2024Fixed Point Diffusion Modelscited by 16
  12. 2024PRoLoRA: Partial Rotation Empowers More Parameter-Efficient LoRAcited by 16
30 more
  1. 2023One Wide Feedforward is All You Needcited by 15
  2. 2024A Comprehensive Survey of Compression Algorithms for Language Modelscited by 15
  3. 2024Adapter-X: A Novel General Parameter-Efficient Fine-Tuning Framework for Visioncited by 13
  4. 2024TroL: Traversal of Layers for Large Language and Vision Modelscited by 13
  5. 2026Training-Free Looped Transformerscited by 13
  6. 2024Dynamic Layer Tying for Parameter-Efficient Transformerscited by 10
  7. 2023Weight-Inherited Distillation for Task-Agnostic BERT Compressioncited by 8
  8. 2024Chain and Causal Attention for Efficient Entity Trackingcited by 8
  9. 2025Layer-wise dynamic rank for compressing large language modelscited by 7
  10. 2022ACORT: A Compact Object Relation Transformer for Parameter Efficient Image Captioningcited by 6
  11. 2025RingFormer: Rethinking Recurrent Transformer with Adaptive Level Signalscited by 6
  12. 2026DeepLoop: Depth Scaling for Looped Transformerscited by 6
  13. 2024Head-wise Shareable Attention for Large Language Modelscited by 4
  14. 2025FlexiGPT: Pruning and Extending Large Language Models with Low-Rank Weight Sharingcited by 4
  15. 2023DSFormer: Effective Compression of Text-Transformers by Dense-Sparse Weight Factorizationcited by 3
  16. 2024DE3-BERT: Distance-Enhanced Early Exiting for BERT based on Prototypical Networkscited by 3
  17. 2025Beyond Parameters: Exploring Virtual Logic Depth for Scaling Lawscited by 3
  18. 2023Zero-Shot Sharpness-Aware Quantization for Pre-trained Language Modelscited by 2
  19. 2026AGI Maze as a Benchmark Framework for World-Modeling Agentscited by 2
  20. 2023PartialFormer: Modeling Part Instead of Whole for Machine Translationcited by 1
  21. 2025Merging Feed-Forward Sublayers for Compressed Transformerscited by 1
  22. 2025Rethinking Parameter Sharing as Graph Coloring for Structured Compressioncited by 1
  23. 2026Depth through recurrence: Looped transformers for flow-matching TTScited by 1
  24. 2023Scaling Pre-trained Language Models to Deeper via Parameter-efficient Architecturecited by 0
  25. 2024SPA: Towards A Computational Friendly Cloud-Base and On-Devices Collaboration Seq2seq Personalized Generation with Casual Inferencecited by 0
  26. 2024Computational design of target-specific linear peptide binders with TransformerBetacited by 0
  27. 2025ScaleNet: Scaling up Pretrained Neural Networks with Incremental Parameterscited by 0
  28. 2026SMolLM: Small Language Models Learn Small Molecular Grammarcited by 0
  29. 2026Mobius Learning: Cyclic Depth Folding in Transformerscited by 0
  30. 2026SAPE: Sandwich Adapters for Parameter Efficiency in Large Language Model Fine-Tuningcited by 0