# Multi-layer Representation Fusion for Neural Machine Translation

arXiv: https://arxiv.org/abs/2002.06714 · HTML: https://mlsanity.com/paper/2002.06714/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Feb 2020.

## Built on by (most cited first, 15 of 15)

- [BridgeTower: Building Bridges Between Encoders in Vision-Language Representation Learning](https://mlsanity.com/paper/2206.08657/) (2022)
- [GTrans: Grouping and Fusing Transformer Layers for Neural Machine Translation](https://mlsanity.com/paper/2207.14467/) (2022)
- [Depth-Wise Attention (DWAtt): A Layer Fusion Method for Data-Efficient Classification](https://mlsanity.com/paper/2209.15168/) (2022)
- [Hybrid-Regressive Neural Machine Translation](https://mlsanity.com/paper/2210.10416/) (2022)
- [Learning to Compose Representations of Different Encoder Layers towards Improving Compositional Generalization](https://mlsanity.com/paper/2305.12169/) (2023)
- [ManagerTower: Aggregating the Insights of Uni-Modal Experts for Vision-Language Representation Learning](https://mlsanity.com/paper/2306.00103/) (2023)
- [Layer-wise Representation Fusion for Compositional Generalization](https://mlsanity.com/paper/2307.10799/) (2023)
- [The NiuTrans System for WNGT 2020 Efficiency Task](https://mlsanity.com/paper/2109.08008/) (2021)
- [Enhancing Speech Recognition Decoding via Layer Aggregation](https://mlsanity.com/paper/2203.11325/) (2022)
- [Don't Be So Sure! Boosting ASR Decoding via Confidence Relaxation](https://mlsanity.com/paper/2212.13378/) (2022)
- [To Understand Representation of Layer-aware Sequence Encoders as Multi-order-graph](https://mlsanity.com/paper/2101.06397/) (2021)
- [Residual Tree Aggregation of Layers for Neural Machine Translation](https://mlsanity.com/paper/2107.14590/) (2021)
- [Enhancing Health Mention Classification Performance: A Study on Advancements in Parameter Efficient Tuning](https://mlsanity.com/paper/2504.21685/) (2025)
- [Manager: Aggregating Insights from Unimodal Experts in Two-Tower VLMs and MLLMs](https://mlsanity.com/paper/2506.11515/) (2025)
- [Utilizing Multilingual Encoders to Improve Large Language Models for Low-Resource Languages](https://mlsanity.com/paper/2508.09091/) (2025)

## Abstract (arXiv)

Neural machine translation systems require a number of stacked layers for deep models. But the prediction depends on the sentence representation of the top-most layer with no access to low-level representations. This makes it more difficult to train the model and poses a risk of information loss to prediction. In this paper, we propose a multi-layer representation fusion (MLRF) approach to fusing stacked layers. In particular, we design three fusion functions to learn a better representation from the stack. Experimental results show that our approach yields improvements of 0.92 and 0.56 BLEU points over the strong Transformer baseline on IWSLT German-English and NIST Chinese-English MT tasks respectively. The result is new state-of-the-art in German-English translation.