# Recurrent Stacking of Layers for Compact Neural Machine Translation Models

arXiv: https://arxiv.org/abs/1807.05353 · HTML: https://mlsanity.com/paper/1807.05353/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jul 2018.

## Built on by (most cited first, 30 of 38)

- [Deep Equilibrium Architectures for Inverse Problems in Imaging](https://mlsanity.com/paper/2102.07944/) (2021)
- [Lessons on Parameter Sharing across Layers in Transformers](https://mlsanity.com/paper/2104.06022/) (2021)
- [Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA](https://mlsanity.com/paper/2410.20672/) (2024)
- [A Survey on Model Compression and Acceleration for Pretrained Language Models](https://mlsanity.com/paper/2202.07105/) (2022)
- [Efficient conformer: Progressive downsampling and grouped attention for automatic speech recognition](https://mlsanity.com/paper/2109.01163/) (2021)
- [IndicBART: A Pre-trained Model for Indic Natural Language Generation](https://mlsanity.com/paper/2109.02903/) (2021)
- [On the Theory of Implicit Deep Learning: Global Convergence with Implicit Layers](https://mlsanity.com/paper/2102.07346/) (2021)
- [The Tunnel Effect: Building Data Representations in Deep Neural Networks](https://mlsanity.com/paper/2305.19753/) (2023)
- [An Overview on Language Models: Recent Developments and Outlook](https://mlsanity.com/paper/2303.05759/) (2023)
- [A global convergence theory for deep ReLU implicit networks via over-parameterization](https://mlsanity.com/paper/2110.05645/) (2021)
- [Solve the Loop: Attractor Models for Language and Reasoning](https://mlsanity.com/paper/2605.12466/) (2026)
- [DLO: Dynamic Layer Operation for Efficient Vertical Scaling of LLMs](https://mlsanity.com/paper/2407.11030/) (2024)
- [Exploring Learngene via Stage-wise Weight Sharing for Initializing Variable-sized Models](https://mlsanity.com/paper/2404.16897/) (2024)
- [Streaming parallel transducer beam search with fast-slow cascaded encoders](https://mlsanity.com/paper/2203.15773/) (2022)
- [YANMTT: Yet Another Neural Machine Translation Toolkit](https://mlsanity.com/paper/2108.11126/) (2021)
- [Speeding up Deep Model Training by Sharing Weights and Then Unsharing](https://mlsanity.com/paper/2110.03848/) (2021)
- [From Growing to Looping: A Unified View of Iterative Computation in LLMs](https://mlsanity.com/paper/2602.16490/) (2026)
- [ACORT: A Compact Object Relation Transformer for Parameter Efficient Image Captioning](https://mlsanity.com/paper/2202.05451/) (2022)
- [Increasing Model Capacity for Free: A Simple Strategy for Parameter Efficient Fine-tuning](https://mlsanity.com/paper/2407.01320/) (2024)
- [Deep Equilibrium Models for Video Snapshot Compressive Imaging](https://mlsanity.com/paper/2201.06931/) (2022)
- [Gradient Descent Optimizes Infinite-Depth ReLU Implicit Networks with Linear Widths](https://mlsanity.com/paper/2205.07463/) (2022)
- [An Empirical Study of Leveraging Knowledge Distillation for Compressing Multilingual Neural Machine Translation Models](https://mlsanity.com/paper/2304.09388/) (2023)
- [Zero Token-Driven Deep Thinking in LLMs: Unlocking the Full Potential of Existing Parameters via Cyclic Refinement](https://mlsanity.com/paper/2502.12214/) (2025)
- [Connections between Deep Equilibrium and Sparse Representation Models with Application to Hyperspectral Image Denoising](https://mlsanity.com/paper/2203.15901/) (2022)
- [Stable Invariant Models via Koopman Spectra](https://mlsanity.com/paper/2207.07475/) (2022)
- [Sharing Low Rank Conformer Weights for Tiny Always-On Ambient Speech Recognition Models](https://mlsanity.com/paper/2303.08343/) (2023)
- [Head-wise Shareable Attention for Large Language Models](https://mlsanity.com/paper/2402.11819/) (2024)
- [Subhomogeneous Deep Equilibrium Models](https://mlsanity.com/paper/2403.00720/) (2024)
- [On the optimization and generalization of overparameterized implicit neural networks](https://mlsanity.com/paper/2209.15562/) (2022)
- [Recurrent Stacking of Layers in Neural Networks: An Application to Neural Machine Translation](https://mlsanity.com/paper/2106.10002/) (2021)

## Abstract (arXiv)

In neural machine translation (NMT), the most common practice is to stack a number of recurrent or feed-forward layers in the encoder and the decoder. As a result, the addition of each new layer improves the translation quality significantly. However, this also leads to a significant increase in the number of parameters. In this paper, we propose to share parameters across all the layers thereby leading to a recurrently stacked NMT model. We empirically show that the translation quality of a model that recurrently stacks a single layer 6 times is comparable to the translation quality of a model that stacks 6 separate layers. We also show that using pseudo-parallel corpora by back-translation leads to further significant improvements in translation quality.