mlsanityData to 7 Oct 2026

GTrans: Grouping and Fusing Transformer Layers for Neural Machine Translation

arXiv 2207.14467 ↗First posted 29 Jul 2022cs.CL cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

Transformer structure, stacked by a sequence of encoder and decoder network layers, achieves significant development in neural machine translation. However, vanilla Transformer mainly exploits the top-layer representation, assuming the lower layers provide trivial or redundant information and thus ignoring the bottom-layer feature that is potentially valuable. In this work, we propose the Group-Transformer model (GTrans) that flexibly divides multi-layer representations of both encoder and decoder into different groups and then fuses these group features to generate target words. To corroborate the effectiveness of the proposed method, extensive experiments and analytic experiments are conducted on three bilingual translation benchmarks and two multilingual translation tasks, including the IWLST-14, IWLST-17, LDC, WMT-14 and OPUS-100 benchmark. Experimental and analytical results demonstrate that our model outperforms its Transformer counterparts by a consistent gain. Furthermore, it can be successfully scaled up to 60 encoder layers and 36 decoder layers.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 49 of its 61 references that are in the map. First posted on arXiv 29 Jul 2022.

Builds on

49 of 61 references.

References · 49

  1. 2017Attention Is All You Needcited by 59,932
  2. 2016Densely Connected Convolutional Networkscited by 6,691
  3. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  4. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  6. 2020Longformer: The Long-Document Transformercited by 3,362
  7. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  8. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  9. 2020Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingcited by 2,141
  10. 2020Reformer: The Efficient Transformercited by 1,658
  11. 2020Rethinking Attention with Performerscited by 1,584
  12. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
37 more
  1. 2020On Layer Normalization in the Transformer Architecturecited by 906
  2. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  3. 2017Convolutional Sequence to Sequence Learningcited by 550
  4. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 467
  5. 2019Learning Deep Transformer Models for Machine Translationcited by 309
  6. 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
  7. 2020ReZero is All You Need: Fast Convergence at Large Depthcited by 260
  8. 2019Massively Multilingual Neural Machine Translationcited by 246
  9. 2019Pay Less Attention with Lightweight and Dynamic Convolutionscited by 211
  10. 2020Understanding the Difficulty of Training Transformerscited by 210
  11. 2021Contrastive Learning for Many-to-many Multilingual Neural Machine Translationcited by 108
  12. 2018The Best of Both Worlds: Combining Recent Advances in Neural Machine Translationcited by 101
  13. 2019Improving Deep Transformer with Depth-Scaled Initialization and Merged Attentioncited by 59
  14. 2016A Convolutional Encoder Model for Neural Machine Translationcited by 54
  15. 2020Very Deep Transformers for Neural Machine Translationcited by 50
  16. 2021DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoderscited by 50
  17. 2019Multilingual Neural Machine Translation with Language Clusteringcited by 47
  18. 2018Training Deeper Neural Machine Translation Models with Transparent Attentioncited by 37
  19. 2020Shallow-to-Deep Training for Neural Machine Translationcited by 36
  20. 2021Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Taskcited by 34
  21. 2020Understanding and Improving Encoder Layer Fusion in Sequence-to-Sequence Learningcited by 26
  22. 2018Soft Layer-Specific Multi-Task Summarization with Entailment and Question Generationcited by 25
  23. 2020XLM-T: Scaling up Multilingual Machine Translation with Pretrained Cross-lingual Transformer Encoderscited by 24
  24. 2018Exploiting Deep Representations for Neural Machine Translationcited by 23
  25. 2021Low-Resource Dialogue Summarization with Domain-Agnostic Multi-Source Pretrainingcited by 19
  26. 2020Deep Transformers with Latent Depthcited by 17
  27. 2020Multi-layer Representation Fusion for Neural Machine Translationcited by 15
  28. 2019Multi-Domain Neural Machine Translation with Word-Level Adaptive Layer-wise Domain Mixingcited by 14
  29. 2019Depth Growing for Neural Machine Translationcited by 13
  30. 2019Neural Machine Translation with Explicit Phrase Alignmentcited by 12
  31. 2022Multilingual Neural Machine Translation with Deep Encoder and Multiple Shallow Decoderscited by 12
  32. 2019Dynamic Layer Aggregation for Neural Machine Translation with Routing-by-Agreementcited by 11
  33. 2020Multiscale Collaborative Deep Models for Neural Machine Translationcited by 11
  34. 2019Lipschitz Constrained Parameter Initialization for Deep Transformerscited by 9
  35. 2020Modeling Future Cost for Neural Machine Translationcited by 4
  36. 2017Multi-channel Encoder for Neural Machine Translationcited by 2
  37. 2020Layer-Wise Multi-View Learning for Neural Machine Translationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 5; 2024: 1; 2025: 2; 2026: 1.

10 papers.

  1. 2025Beyond the Surface: Enhancing LLM-as-a-Judge Alignment with Human via Internal Representationscited by 14
  2. 2023Textless Unit-to-Unit training for Many-to-Many Multilingual Speech-to-Speech Translationcited by 11
  3. 2023Improving Long Context Document-Level Machine Translationcited by 8
  4. 2025Beyond Decoder-only: Large Language Models Can be Good Encoders for Machine Translationcited by 8
  5. 2023Adaptive Neural Ranking Framework: Toward Maximized Business Goal for Cascade Ranking Systemscited by 6
  6. 2024Leveraging Large Language Models for Enhanced NLP Task Performance through Knowledge Distillation and Optimized Training Strategiescited by 6
  7. 2022Foreground Guidance and Multi-Layer Feature Fusion for Unsupervised Object Discovery with Transformerscited by 4
  8. 2023N-Gram Nearest Neighbor Machine Translationcited by 1
  9. 2023On Search Strategies for Document-Level Neural Machine Translationcited by 0
  10. 2026Data Fusion and Contrastive Alignment for Unconstrained IR Molecular Structure Elucidationcited by 0