mlsanityData to 14 Oct 2026

Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA

arXiv 2410.20672 ↗First posted 28 Oct 2024cs.CL cs.LGcited by 78

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) are expensive to deploy. Parameter sharing offers a possible path towards reducing their size and cost, but its effectiveness in modern LLMs remains fairly limited. In this work, we revisit "layer tying" as form of parameter sharing in Transformers, and introduce novel methods for converting existing LLMs into smaller "Recursive Transformers" that share parameters across layers, with minimal loss of performance. Here, our Recursive Transformers are efficiently initialized from standard pretrained Transformers, but only use a single block of unique layers that is then repeated multiple times in a loop. We further improve performance by introducing Relaxed Recursive Transformers that add flexibility to the layer tying constraint via depth-wise low-rank adaptation (LoRA) modules, yet still preserve the compactness of the overall model. We show that our recursive models (e.g., recursive Gemma 1B) outperform both similar-sized vanilla pretrained models (such as TinyLlama 1.1B and Pythia 1B) and knowledge distillation baselines -- and can even recover most of the performance of the original "full-size" model (e.g., Gemma 2B with no shared parameters). Finally, we propose Continuous Depth-wise Batching, a promising new inference paradigm enabled by the Recursive Transformer when paired with early exiting. In a theoretical analysis, we show that this has the potential to lead to significant (2-3x) gains in inference throughput.

Key facts. Cited by 78 papers in the map (data through 14 Oct 2026). Builds on 86 of its 111 references that are in the map. First posted on arXiv 28 Oct 2024.

Builds on

86 of 111 references.

References · 86

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  9. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  10. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  11. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  12. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
38 more
  1. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  2. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  3. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  4. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  5. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  6. 2020GLU Variants Improve Transformercited by 1,804
  7. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  8. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  9. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  10. 2019Root Mean Square Layer Normalizationcited by 1,480
  11. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  12. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  13. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  14. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  15. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  16. 2016Sequence-Level Knowledge Distillationcited by 801
  17. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  18. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  19. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 689
  20. 2018Universal Transformerscited by 647
  21. 2024TinyLlama: An Open-Source Small Language Modelcited by 618
  22. 2019Deep Equilibrium Modelscited by 615
  23. 2022Efficiently Scaling Transformer Inferencecited by 539
  24. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 506
  25. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 439
  26. 2018Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Networkcited by 420
  27. 2022Confident Adaptive Language Modelingcited by 310
  28. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 307
  29. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 269
  30. 2023Think before you speak: Training Language Models With Pause Tokenscited by 264
  31. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 258
  32. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 252
  33. 2019What's Hidden in a Randomly Weighted Neural Network?cited by 237
  34. 2023Looped Transformers as Programmable Computerscited by 237
  35. 2019Depth-Adaptive Transformercited by 236
  36. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 223
  37. 2019A Constructive Prediction of the Generalization Error Across Scalescited by 210
  38. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 209

50 most cited of 86 All 86 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 19; 2026: 58.

78 papers.

Sort:

By area:

  1. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 415
  2. 2025Scaling Latent Reasoning via Looped Language Modelscited by 165
  3. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 133
  4. 2025A Survey on Latent Reasoningcited by 81
  5. 2026LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulationcited by 61
  6. 2026Parcae: Scaling Laws For Stable Looped Language Modelscited by 55
  7. 2025Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrencecited by 47
  8. 2025Enabling Agents to Communicate Entirely in Latent Spacecited by 34
  9. 2025Coevolutionary Continuous Discrete Diffusion: Make Your Diffusion Language Model a Latent Reasonercited by 32
  10. 2025Encode, Think, Decode: Scaling test-time reasoning with recursive latent thoughtscited by 24
  11. 2026Generative Recursive Reasoningcited by 20
  12. 2026SMELT: Scaling Laws for Compute-Matched MoE Looped Transformerscited by 18
38 more
  1. 2026LoopUS: Recasting Pretrained LLMs into Looped Latent Refinement Modelscited by 15
  2. 2026Training-Free Looped Transformerscited by 15
  3. 2026Solve the Loop: Attractor Models for Language and Reasoningcited by 13
  4. 2026Depth-Recurrent Attention Mixtures: Giving Latent Reasoning the Attention it Deservescited by 12
  5. 2026SpiralFormer: Looped Transformers Can Learn Hierarchical Dependencies via Multi-Resolution Recursioncited by 12
  6. 2026Hyperloop Transformerscited by 12
  7. 2025ReXMoE: Reusing Experts with Minimal Overhead in Mixture-of-Expertscited by 10
  8. 2026LoopMoE: Unifying Iterative Computation with Mixture-of-Experts for Language Modelingcited by 10
  9. 2025MeSH: Memory-as-State-Highways for Recursive Transformerscited by 8
  10. 2025SCOUT: Teaching Pre-trained Language Models to Enhance Reasoning via Flow Chain-of-Thoughtcited by 7
  11. 2025AbbIE: Autoregressive Block-Based Iterative Encoder for Efficient Sequence Modelingcited by 7
  12. 2025Efficient Parallel Samplers for Recurrent-Depth Models and Their Connection to Diffusion Language Modelscited by 7
  13. 2026From Growing to Looping: A Unified View of Iterative Computation in LLMscited by 7
  14. 2026LT2: Linear-Time Looped Transformerscited by 7
  15. 2026Depth-adaptive Inference of Looped Language Models via Continuous Depth Batchingcited by 6
  16. 2026Retrofitting Recurrent Depth into a Pretrained Language Model: Installation, Extrapolation, Transfer, and Retention at Two Parameter Budgetscited by 6
  17. 2025Zero Token-Driven Deep Thinking in LLMs: Unlocking the Full Potential of Existing Parameters via Cyclic Refinementcited by 5
  18. 2026Looping Back to Move Forward: Recursive Transformers for Efficient and Flexible Large Multimodal Modelscited by 5
  19. 2026Fixed-Point Masked Generative Modelingcited by 5
  20. 2026Tying the Loop -- Tied Expert Layers in Mixture-of-Experts Language Modelscited by 4
  21. 2026LoopCoder-v2: Only Loop Once for Efficient Test-Time Computation Scalingcited by 4
  22. 2026Gated Recurrent Transformers: Expressive Depth through Recurrent Modulationcited by 4
  23. 2024ASLoRA: Adaptive Sharing Low-Rank Adaptation Across Layerscited by 3
  24. 2025DeltaLLM: Compress LLMs with Low-Rank Deltas between Shared Weightscited by 3
  25. 2026Expert-Choice Routing Enables Adaptive Computation in Diffusion Language Modelscited by 3
  26. 2026How Transparent is DiffusionGemma?cited by 3
  27. 2026MoRE: Mixture of Reused Expertscited by 3
  28. 2025Hyper-SET: Designing Transformers via Hyperspherical Energy Minimizationcited by 2
  29. 2025Improving Recursive Transformers with Mixture of LoRAscited by 2
  30. 2026Recursive Scaling in Masked Diffusion Modelscited by 2
  31. 2026Ouroboros: Dynamic Weight Generation for Recursive Transformers via Input-Conditioned LoRA Modulationcited by 1
  32. 2026UniPool: A Globally Shared Expert Pool for Mixture-of-Expertscited by 1
  33. 2026Scratchpad Patching: Decoupling Compute from Patch Size in Byte-Level Language Modelscited by 1
  34. 2026Steering Recurrent Reasoners at Inference Time with Readout Feedbackcited by 1
  35. 2026Right Direction, Wrong Step: Geometric Analysis of Finite-Step Failure in Looped Transformerscited by 1
  36. 2026Prediction Dynamics in Depth-Recurrent Language Modelscited by 1
  37. 2026Improving Test-Time Scaling with Adaptive Looped Transformerscited by 1
  38. 2026How to Loop MoE: Flatten the Experts, Untie the Attentioncited by 1

50 most cited of 78 All 78 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Aletheia: Gradient-Guided Layer Selection for Efficient LoRA Fine-Tuning Across Architectures
  2. 2025Equivalence of Context and Parameter Updates in Modern Transformer Blocks
  3. 2024Higher Layers Need More LoRA Experts
  4. 2024FinerCut: Finer-grained Interpretable Layer Pruning for Large Language Models
  5. 2025Speed Always Wins: A Survey on Efficient Architectures for Large Language Models
  6. 2024DLO: Dynamic Layer Operation for Efficient Vertical Scaling of LLMs
  7. 2026Sparse Layers are Critical to Scaling Looped Language Models
  8. 2023A Comparative Analysis of Task-Agnostic Distillation Methods for Compressing Transformer Language Models