mlsanityData to 7 Oct 2026

Dynamic Layer Tying for Parameter-Efficient Transformers

arXiv 2401.12819 ↗First posted 23 Jan 2024cs.LG cs.AIcited by 10

Not on the map. Why?

Abstract (arXiv)

In the pursuit of reducing the number of trainable parameters in deep transformer networks, we employ Reinforcement Learning to dynamically select layers during training and tie them together. Every few iterations, the RL agent is asked whether to train each layer i independently or to copy the weights of a previous layer j<i. This facilitates weight sharing, reduces the number of trainable parameters, and also serves as an effective regularization technique. Experimental evaluations validate that our model modestly outperforms the baseline transformer model with regard to perplexity and drastically reduces the number of trainable parameters. In particular, the memory consumption during training is up to one order of magnitude less than the conventional training method.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 49 of its 57 references that are in the map. First posted on arXiv 23 Jan 2024.

Builds on

49 of 57 references.

References · 49

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2017Proximal Policy Optimization Algorithmscited by 14,441
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  11. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  12. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
37 more
  1. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  2. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  3. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  4. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  5. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  6. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  7. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  8. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  9. 2016An overview of gradient descent optimization algorithmscited by 1,021
  10. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  11. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 831
  12. 2019Are Sixteen Heads Really Better than One?cited by 803
  13. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  14. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
  15. 2018Rethinking the Value of Network Pruningcited by 595
  16. 2019The State of Sparsity in Deep Neural Networkscited by 471
  17. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  18. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 377
  19. 2016Designing Neural Network Architectures using Reinforcement Learningcited by 277
  20. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
  21. 2021Training Neural Networks with Fixed Sparse Maskscited by 212
  22. 2022A Fast Post-Training Pruning Framework for Transformerscited by 161
  23. 2020On the Effect of Dropping Layers of Pre-trained Transformer Modelscited by 118
  24. 2022The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Modelscited by 115
  25. 2020When BERT Plays the Lottery, All Tickets Are Winningcited by 114
  26. 2021Luna: Linear Unified Nested Attentioncited by 96
  27. 2021Lessons on Parameter Sharing across Layers in Transformerscited by 85
  28. 2020EarlyBERT: Efficient BERT Training via Early-bird Lottery Ticketscited by 79
  29. 2021Cross-Attention is All You Need: Adapting Pretrained Transformers for Machine Translationcited by 74
  30. 2021Subformer: Exploring Weight Sharing for Parameter Efficiency in Generative Transformerscited by 42
  31. 2019GraphNAS: Graph Neural Architecture Search with Reinforcement Learningcited by 41
  32. 2019Sharing Attention Weights for Fast Transformercited by 34
  33. 2021Leveraging redundancy in attention with Reuse Transformerscited by 33
  34. 2019Learning an Adaptive Learning Rate Schedulecited by 29
  35. 2022Efficient Fine-Tuning of BERT Models on the Edgecited by 29
  36. 2020Dissecting Lottery Ticket Transformers: Structural and Behavioral Study of Sparse Neural Machine Translationcited by 8
  37. 2022Dependency Parsing with Backtracking using Deep Reinforcement Learningcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 8; 2026: 1.

10 papers.

  1. 2024Basis Sharing: Cross-Layer Parameter Sharing for Large Language Model Compressioncited by 34
  2. 2025Encode, Think, Decode: Scaling test-time reasoning with recursive latent thoughtscited by 23
  3. 2026SpiralFormer: Looped Transformers Can Learn Hierarchical Dependencies via Multi-Resolution Recursioncited by 9
  4. 2025MeSH: Memory-as-State-Highways for Recursive Transformerscited by 8
  5. 2025Efficient Parallel Samplers for Recurrent-Depth Models and Their Connection to Diffusion Language Modelscited by 7
  6. 2025Layer-wise dynamic rank for compressing large language modelscited by 6
  7. 2025Beyond Parameters: Exploring Virtual Logic Depth for Scaling Lawscited by 2
  8. 2025Merging Feed-Forward Sublayers for Compressed Transformerscited by 1
  9. 2025Rethinking Parameter Sharing as Graph Coloring for Structured Compressioncited by 1
  10. 2025KernelDNA: Dynamic Kernel Sharing via Decoupled Naive Adapterscited by 0