# Dynamic Layer Aggregation for Neural Machine Translation with Routing-by-Agreement

arXiv: https://arxiv.org/abs/1902.05770 · HTML: https://mlsanity.com/paper/1902.05770/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Feb 2019.

## Built on by (most cited first, 11 of 11)

- [BridgeTower: Building Bridges Between Encoders in Vision-Language Representation Learning](https://mlsanity.com/paper/2206.08657/) (2022)
- [Turning Fixed to Adaptive: Integrating Post-Evaluation into Simultaneous Machine Translation](https://mlsanity.com/paper/2210.11900/) (2022)
- [Residual Matrix Transformers: Scaling the Size of the Residual Stream](https://mlsanity.com/paper/2506.22696/) (2025)
- [GTrans: Grouping and Fusing Transformer Layers for Neural Machine Translation](https://mlsanity.com/paper/2207.14467/) (2022)
- [Depth-Wise Attention (DWAtt): A Layer Fusion Method for Data-Efficient Classification](https://mlsanity.com/paper/2209.15168/) (2022)
- [An Algorithm for Routing Vectors in Sequences](https://mlsanity.com/paper/2211.11754/) (2022)
- [Grounding Commands for Autonomous Vehicles via Layer Fusion with Region-specific Dynamic Layer Attention](https://mlsanity.com/paper/2203.06822/) (2022)
- [Learning to Compose Representations of Different Encoder Layers towards Improving Compositional Generalization](https://mlsanity.com/paper/2305.12169/) (2023)
- [β-CapsNet: Learning Disentangled Representation for CapsNet by Information Bottleneck](https://mlsanity.com/paper/2209.05239/) (2022)
- [Residual Tree Aggregation of Layers for Neural Machine Translation](https://mlsanity.com/paper/2107.14590/) (2021)
- [Two Heads are Better than One: Distilling Large Language Model Features Into Small Models with Feature Decomposition and Mixture](https://mlsanity.com/paper/2511.07110/) (2025)

## Abstract (arXiv)

With the promising progress of deep neural networks, layer aggregation has been used to fuse information across layers in various fields, such as computer vision and machine translation. However, most of the previous methods combine layers in a static fashion in that their aggregation strategy is independent of specific hidden states. Inspired by recent progress on capsule networks, in this paper we propose to use routing-by-agreement strategies to aggregate layers dynamically. Specifically, the algorithm learns the probability of a part (individual layer representations) assigned to a whole (aggregated representations) in an iterative way and combines parts accordingly. We implement our algorithm on top of the state-of-the-art neural machine translation model TRANSFORMER and conduct experiments on the widely-used WMT14 English-German and WMT17 Chinese-English translation datasets. Experimental results across language pairs show that the proposed approach consistently outperforms the strong baseline model and a representative static aggregation model.