# Residual Tree Aggregation of Layers for Neural Machine Translation

arXiv: https://arxiv.org/abs/2107.14590 · HTML: https://mlsanity.com/paper/2107.14590/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 24 references that are in the map.
- First posted on arXiv 19 Jul 2021.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [Convolutional Sequence to Sequence Learning](https://mlsanity.com/paper/1705.03122/) (2017)
- [Deep Layer Aggregation](https://mlsanity.com/paper/1707.06484/) (2017)
- [The Evolved Transformer](https://mlsanity.com/paper/1901.11117/) (2019)
- [Tensor2Tensor for Neural Machine Translation](https://mlsanity.com/paper/1803.07416/) (2018)
- [Exploiting Deep Representations for Neural Machine Translation](https://mlsanity.com/paper/1810.10181/) (2018)
- [Multi-layer Representation Fusion for Neural Machine Translation](https://mlsanity.com/paper/2002.06714/) (2020)
- [Dynamic Layer Aggregation for Neural Machine Translation with Routing-by-Agreement](https://mlsanity.com/paper/1902.05770/) (2019)
- [Dense Information Flow for Neural Machine Translation](https://mlsanity.com/paper/1806.00722/) (2018)

## Abstract (arXiv)

Although attention-based Neural Machine Translation has achieved remarkable progress in recent layers, it still suffers from issue of making insufficient use of the output of each layer. In transformer, it only uses the top layer of encoder and decoder in the subsequent process, which makes it impossible to take advantage of the useful information in other layers. To address this issue, we propose a residual tree aggregation of layers for Transformer(RTAL), which helps to fuse information across layers. Specifically, we try to fuse the information across layers by constructing a post-order binary tree. In additional to the last node, we add the residual connection to the process of generating child nodes. Our model is based on the Neural Machine Translation model Transformer and we conduct our experiments on WMT14 English-to-German and WMT17 English-to-France translation tasks. Experimental results across language pairs show that the proposed approach outperforms the strong baseline model significantly