# GTrans: Grouping and Fusing Transformer Layers for Neural Machine Translation

arXiv: https://arxiv.org/abs/2207.14467 · HTML: https://mlsanity.com/paper/2207.14467/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 61 references that are in the map.
- First posted on arXiv 29 Jul 2022.

## Builds on (most cited first, 30 of 49)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [Beyond English-Centric Multilingual Machine Translation](https://mlsanity.com/paper/2010.11125/) (2020)
- [Convolutional Sequence to Sequence Learning](https://mlsanity.com/paper/1705.03122/) (2017)
- [Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation](https://mlsanity.com/paper/1611.04558/) (2016)
- [Learning Deep Transformer Models for Machine Translation](https://mlsanity.com/paper/1906.01787/) (2019)
- [Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation](https://mlsanity.com/paper/2004.11867/) (2020)
- [ReZero is All You Need: Fast Convergence at Large Depth](https://mlsanity.com/paper/2003.04887/) (2020)
- [Massively Multilingual Neural Machine Translation](https://mlsanity.com/paper/1903.00089/) (2019)
- [Pay Less Attention with Lightweight and Dynamic Convolutions](https://mlsanity.com/paper/1901.10430/) (2019)
- [Understanding the Difficulty of Training Transformers](https://mlsanity.com/paper/2004.08249/) (2020)
- [Contrastive Learning for Many-to-many Multilingual Neural Machine Translation](https://mlsanity.com/paper/2105.09501/) (2021)
- [The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation](https://mlsanity.com/paper/1804.09849/) (2018)
- [Improving Deep Transformer with Depth-Scaled Initialization and Merged Attention](https://mlsanity.com/paper/1908.11365/) (2019)
- [A Convolutional Encoder Model for Neural Machine Translation](https://mlsanity.com/paper/1611.02344/) (2016)
- [Very Deep Transformers for Neural Machine Translation](https://mlsanity.com/paper/2008.07772/) (2020)
- [DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders](https://mlsanity.com/paper/2106.13736/) (2021)
- [Multilingual Neural Machine Translation with Language Clustering](https://mlsanity.com/paper/1908.09324/) (2019)
- [Training Deeper Neural Machine Translation Models with Transparent Attention](https://mlsanity.com/paper/1808.07561/) (2018)

## Built on by (most cited first, 10 of 10)

- [Beyond the Surface: Enhancing LLM-as-a-Judge Alignment with Human via Internal Representations](https://mlsanity.com/paper/2508.03550/) (2025)
- [Textless Unit-to-Unit training for Many-to-Many Multilingual Speech-to-Speech Translation](https://mlsanity.com/paper/2308.01831/) (2023)
- [Improving Long Context Document-Level Machine Translation](https://mlsanity.com/paper/2306.05183/) (2023)
- [Beyond Decoder-only: Large Language Models Can be Good Encoders for Machine Translation](https://mlsanity.com/paper/2503.06594/) (2025)
- [Adaptive Neural Ranking Framework: Toward Maximized Business Goal for Cascade Ranking Systems](https://mlsanity.com/paper/2310.10462/) (2023)
- [Leveraging Large Language Models for Enhanced NLP Task Performance through Knowledge Distillation and Optimized Training Strategies](https://mlsanity.com/paper/2402.09282/) (2024)
- [Foreground Guidance and Multi-Layer Feature Fusion for Unsupervised Object Discovery with Transformers](https://mlsanity.com/paper/2210.13053/) (2022)
- [N-Gram Nearest Neighbor Machine Translation](https://mlsanity.com/paper/2301.12866/) (2023)
- [On Search Strategies for Document-Level Neural Machine Translation](https://mlsanity.com/paper/2306.05116/) (2023)
- [Data Fusion and Contrastive Alignment for Unconstrained IR Molecular Structure Elucidation](https://mlsanity.com/paper/2607.26164/) (2026)

## Abstract (arXiv)

Transformer structure, stacked by a sequence of encoder and decoder network layers, achieves significant development in neural machine translation. However, vanilla Transformer mainly exploits the top-layer representation, assuming the lower layers provide trivial or redundant information and thus ignoring the bottom-layer feature that is potentially valuable. In this work, we propose the Group-Transformer model (GTrans) that flexibly divides multi-layer representations of both encoder and decoder into different groups and then fuses these group features to generate target words. To corroborate the effectiveness of the proposed method, extensive experiments and analytic experiments are conducted on three bilingual translation benchmarks and two multilingual translation tasks, including the IWLST-14, IWLST-17, LDC, WMT-14 and OPUS-100 benchmark. Experimental and analytical results demonstrate that our model outperforms its Transformer counterparts by a consistent gain. Furthermore, it can be successfully scaled up to 60 encoder layers and 36 decoder layers.