# DTMT: A Novel Deep Transition Architecture for Neural Machine Translation

arXiv: https://arxiv.org/abs/1812.07807 · HTML: https://mlsanity.com/paper/1812.07807/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Dec 2018.

## Built on by (most cited first, 13 of 13)

- [Selective Knowledge Distillation for Neural Machine Translation](https://mlsanity.com/paper/2105.12967/) (2021)
- [Prevent the Language Model from being Overconfident in Neural Machine Translation](https://mlsanity.com/paper/2105.11098/) (2021)
- [Modeling Bilingual Conversational Characteristics for Neural Chat Translation](https://mlsanity.com/paper/2107.11164/) (2021)
- [MSCTD: A Multimodal Sentiment Chat Translation Dataset](https://mlsanity.com/paper/2202.13645/) (2022)
- [Towards Making the Most of Dialogue Characteristics for Neural Chat Translation](https://mlsanity.com/paper/2109.00668/) (2021)
- [WeChat Neural Machine Translation Systems for WMT21](https://mlsanity.com/paper/2108.02401/) (2021)
- [Confidence Based Bidirectional Global Context Aware Training Framework for Neural Machine Translation](https://mlsanity.com/paper/2202.13663/) (2022)
- [Scheduled Multi-task Learning for Neural Chat Translation](https://mlsanity.com/paper/2205.03766/) (2022)
- [Conditional Bilingual Mutual Information Based Adaptive Training for Neural Machine Translation](https://mlsanity.com/paper/2203.02951/) (2022)
- [Bilingual Mutual Information Based Adaptive Training for Neural Machine Translation](https://mlsanity.com/paper/2105.12523/) (2021)
- [DC-MBR: Distributional Cooling for Minimum Bayesian Risk Decoding](https://mlsanity.com/paper/2212.04205/) (2022)
- [BJTU-WeChat's Systems for the WMT22 Chat Translation Task](https://mlsanity.com/paper/2211.15009/) (2022)
- [A Global Context Mechanism for Sequence Labeling](https://mlsanity.com/paper/2305.19928/) (2023)

## Abstract (arXiv)

Past years have witnessed rapid developments in Neural Machine Translation (NMT). Most recently, with advanced modeling and training techniques, the RNN-based NMT (RNMT) has shown its potential strength, even compared with the well-known Transformer (self-attentional) model. Although the RNMT model can possess very deep architectures through stacking layers, the transition depth between consecutive hidden states along the sequential axis is still shallow. In this paper, we further enhance the RNN-based NMT through increasing the transition depth between consecutive hidden states and build a novel Deep Transition RNN-based Architecture for Neural Machine Translation, named DTMT. This model enhances the hidden-to-hidden transition with multiple non-linear transformations, as well as maintains a linear transformation path throughout this deep transition by the well-designed linear transformation mechanism to alleviate the gradient vanishing problem. Experiments show that with the specially designed deep transition modules, our DTMT can achieve remarkable improvements on translation quality. Experimental results on Chinese->English translation task show that DTMT can outperform the Transformer model by +2.09 BLEU points and achieve the best results ever reported in the same dataset. On WMT14 English->German and English->French translation tasks, DTMT shows superior quality to the state-of-the-art NMT systems, including the Transformer and the RNMT+.