# Exploiting Deep Representations for Neural Machine Translation

arXiv: https://arxiv.org/abs/1810.10181 · HTML: https://mlsanity.com/paper/1810.10181/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Oct 2018.

## Built on by (most cited first, 23 of 23)

- [An Empirical Study of Training End-to-End Vision-and-Language Transformers](https://mlsanity.com/paper/2111.02387/) (2021)
- [The Curse of Depth in Large Language Models](https://mlsanity.com/paper/2502.05795/) (2025)
- [BridgeTower: Building Bridges Between Encoders in Vision-Language Representation Learning](https://mlsanity.com/paper/2206.08657/) (2022)
- [Subformer: Exploring Weight Sharing for Parameter Efficiency in Generative Transformers](https://mlsanity.com/paper/2101.00234/) (2021)
- [PingAn-VCGroup's Solution for ICDAR 2021 Competition on Scientific Literature Parsing Task B: Table Recognition to HTML](https://mlsanity.com/paper/2105.01848/) (2021)
- [Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LN](https://mlsanity.com/paper/2412.13795/) (2024)
- [Rethinking Skip Connection with Layer Normalization in Transformers and ResNets](https://mlsanity.com/paper/2105.07205/) (2021)
- [DecoderLens: Layerwise Interpretation of Encoder-Decoder Transformers](https://mlsanity.com/paper/2310.03686/) (2023)
- [B2T Connection: Serving Stability and Performance in Deep Transformers](https://mlsanity.com/paper/2206.00330/) (2022)
- [Text Compression-aided Transformer Encoding](https://mlsanity.com/paper/2102.05951/) (2021)
- [Residual Matrix Transformers: Scaling the Size of the Residual Stream](https://mlsanity.com/paper/2506.22696/) (2025)
- [GTrans: Grouping and Fusing Transformer Layers for Neural Machine Translation](https://mlsanity.com/paper/2207.14467/) (2022)
- [Beyond Decoder-only: Large Language Models Can be Good Encoders for Machine Translation](https://mlsanity.com/paper/2503.06594/) (2025)
- [Auto-Compressing Networks](https://mlsanity.com/paper/2506.09714/) (2025)
- [Recurrent multiple shared layers in Depth for Neural Machine Translation](https://mlsanity.com/paper/2108.10417/) (2021)
- [Learning to Compose Representations of Different Encoder Layers towards Improving Compositional Generalization](https://mlsanity.com/paper/2305.12169/) (2023)
- [Layer-wise Representation Fusion for Compositional Generalization](https://mlsanity.com/paper/2307.10799/) (2023)
- [Don't Be So Sure! Boosting ASR Decoding via Confidence Relaxation](https://mlsanity.com/paper/2212.13378/) (2022)
- [Lightweight Adaptive Feature Composition for Heterogeneous Downstream Adaptation of Wireless Foundation Models](https://mlsanity.com/paper/2606.10277/) (2026)
- [Residual Tree Aggregation of Layers for Neural Machine Translation](https://mlsanity.com/paper/2107.14590/) (2021)
- [Integrating Pre-trained Language Model into Neural Machine Translation](https://mlsanity.com/paper/2310.19680/) (2023)
- [Exploring the traditional NMT model and Large Language Model for chat translation](https://mlsanity.com/paper/2409.16331/) (2024)
- [Data Fusion and Contrastive Alignment for Unconstrained IR Molecular Structure Elucidation](https://mlsanity.com/paper/2607.26164/) (2026)

## Abstract (arXiv)

Advanced neural machine translation (NMT) models generally implement encoder and decoder as multiple layers, which allows systems to model complex functions and capture complicated linguistic structures. However, only the top layers of encoder and decoder are leveraged in the subsequent process, which misses the opportunity to exploit the useful information embedded in other layers. In this work, we propose to simultaneously expose all of these signals with layer aggregation and multi-layer attention mechanisms. In addition, we introduce an auxiliary regularization term to encourage different layers to capture diverse information. Experimental results on widely-used WMT14 English-German and WMT17 Chinese-English translation data demonstrate the effectiveness and universality of the proposed approach.