mlsanityData to 13 Oct 2026

Recurrent Stacking of Layers for Compact Neural Machine Translation Models

arXiv 1807.05353 ↗First posted 14 Jul 2018cs.CLcited by 38

Not on the map. Why?

Abstract (arXiv)

In neural machine translation (NMT), the most common practice is to stack a number of recurrent or feed-forward layers in the encoder and the decoder. As a result, the addition of each new layer improves the translation quality significantly. However, this also leads to a significant increase in the number of parameters. In this paper, we propose to share parameters across all the layers thereby leading to a recurrently stacked NMT model. We empirically show that the translation quality of a model that recurrently stacks a single layer 6 times is comparable to the translation quality of a model that stacks 6 separate layers. We also show that using pseudo-parallel corpora by back-translation leads to further significant improvements in translation quality.

Key facts. Cited by 38 papers in the map (data through 13 Oct 2026). First posted on arXiv 14 Jul 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 9; 2022: 9; 2023: 7; 2024: 6; 2025: 1; 2026: 6.

38 papers.

Sort:

By area:

  1. 2021Deep Equilibrium Architectures for Inverse Problems in Imagingcited by 126
  2. 2021Lessons on Parameter Sharing across Layers in Transformerscited by 85
  3. 2024Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRAcited by 74
  4. 2022A Survey on Model Compression and Acceleration for Pretrained Language Modelscited by 45
  5. 2021Efficient conformer: Progressive downsampling and grouped attention for automatic speech recognitioncited by 42
  6. 2021IndicBART: A Pre-trained Model for Indic Natural Language Generationcited by 39
  7. 2021On the Theory of Implicit Deep Learning: Global Convergence with Implicit Layerscited by 35
  8. 2023The Tunnel Effect: Building Data Representations in Deep Neural Networkscited by 34
  9. 2023An Overview on Language Models: Recent Developments and Outlookcited by 22
  10. 2021A global convergence theory for deep ReLU implicit networks via over-parameterizationcited by 17
  11. 2026Solve the Loop: Attractor Models for Language and Reasoningcited by 11
  12. 2024DLO: Dynamic Layer Operation for Efficient Vertical Scaling of LLMscited by 10
26 more
  1. 2024Exploring Learngene via Stage-wise Weight Sharing for Initializing Variable-sized Modelscited by 9
  2. 2022Streaming parallel transducer beam search with fast-slow cascaded encoderscited by 8
  3. 2021YANMTT: Yet Another Neural Machine Translation Toolkitcited by 7
  4. 2021Speeding up Deep Model Training by Sharing Weights and Then Unsharingcited by 7
  5. 2022ACORT: A Compact Object Relation Transformer for Parameter Efficient Image Captioningcited by 6
  6. 2024Increasing Model Capacity for Free: A Simple Strategy for Parameter Efficient Fine-tuningcited by 6
  7. 2026From Growing to Looping: A Unified View of Iterative Computation in LLMscited by 6
  8. 2022Deep Equilibrium Models for Video Snapshot Compressive Imagingcited by 5
  9. 2022Gradient Descent Optimizes Infinite-Depth ReLU Implicit Networks with Linear Widthscited by 5
  10. 2023An Empirical Study of Leveraging Knowledge Distillation for Compressing Multilingual Neural Machine Translation Modelscited by 5
  11. 2025Zero Token-Driven Deep Thinking in LLMs: Unlocking the Full Potential of Existing Parameters via Cyclic Refinementcited by 5
  12. 2022Connections between Deep Equilibrium and Sparse Representation Models with Application to Hyperspectral Image Denoisingcited by 4
  13. 2022Stable Invariant Models via Koopman Spectracited by 4
  14. 2023Sharing Low Rank Conformer Weights for Tiny Always-On Ambient Speech Recognition Modelscited by 4
  15. 2024Head-wise Shareable Attention for Large Language Modelscited by 4
  16. 2024Subhomogeneous Deep Equilibrium Modelscited by 3
  17. 2022On the optimization and generalization of overparameterized implicit neural networkscited by 2
  18. 2021Recurrent Stacking of Layers in Neural Networks: An Application to Neural Machine Translationcited by 1
  19. 2023Predicting Human Translation Difficulty with Neural Machine Translationcited by 1
  20. 2026Looped Diffusion Language Modelscited by 1
  21. 2026Looped Transformers with Source-Centered State Evolutioncited by 1
  22. 2022FusionFormer: Fusing Operations in Transformer for Efficient Streaming Speech Recognitioncited by 0
  23. 2023Revisiting Implicit Models: Sparsity Trade-offs Capability in Weight-tied Model for Vision Taskscited by 0
  24. 2023IDKM: Memory Efficient Neural Network Quantization via Implicit, Differentiable k-Meanscited by 0
  25. 2026RePAIR: Predictive Self-Supervised Representation Learning in Chesscited by 0
  26. 2026Loop Dropout: Regularizing Shared Updates in Looped Language Modelscited by 0