# Depth Growing for Neural Machine Translation

arXiv: https://arxiv.org/abs/1907.01968 · HTML: https://mlsanity.com/paper/1907.01968/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Jul 2019.

## Built on by (most cited first, 13 of 13)

- [R-Drop: Regularized Dropout for Neural Networks](https://mlsanity.com/paper/2106.14448/) (2021)
- [Taming Sparsely Activated Transformer with Stochastic Experts](https://mlsanity.com/paper/2110.04260/) (2021)
- [A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond](https://mlsanity.com/paper/2204.09269/) (2022)
- [ODE Transformer: An Ordinary Differential Equation-Inspired Model for Sequence Generation](https://mlsanity.com/paper/2203.09176/) (2022)
- [ODE Transformer: An Ordinary Differential Equation-Inspired Model for Neural Machine Translation](https://mlsanity.com/paper/2104.02308/) (2021)
- [GTrans: Grouping and Fusing Transformer Layers for Neural Machine Translation](https://mlsanity.com/paper/2207.14467/) (2022)
- [AMOM: Adaptive Masking over Masking for Conditional Masked Language Model](https://mlsanity.com/paper/2303.07457/) (2023)
- [Recurrent multiple shared layers in Depth for Neural Machine Translation](https://mlsanity.com/paper/2108.10417/) (2021)
- [Understanding Parameter Sharing in Transformers](https://mlsanity.com/paper/2306.09380/) (2023)
- [RenewNAT: Renewing Potential Translation for Non-Autoregressive Transformer](https://mlsanity.com/paper/2303.07665/) (2023)
- [ClaimFlow: Tracing the Evolution of Scientific Claims in NLP](https://mlsanity.com/paper/2603.16073/) (2026)
- [Optimizing Deep Transformers for Chinese-Thai Low-Resource Translation](https://mlsanity.com/paper/2212.12662/) (2022)
- [Enhancing Context Through Contrast](https://mlsanity.com/paper/2401.03314/) (2024)

## Abstract (arXiv)

While very deep neural networks have shown effectiveness for computer vision and text classification applications, how to increase the network depth of neural machine translation (NMT) models for better translation quality remains a challenging problem. Directly stacking more blocks to the NMT model results in no improvement and even reduces performance. In this work, we propose an effective two-stage approach with three specially designed components to construct deeper NMT models, which result in significant improvements over the strong Transformer baselines on WMT14 English→German and English→French translation tasksfootnoteOur code is available at urlhttps://github.com/apeterswu/DepthGrowingNMT.