mlsanityData to 14 Oct 2026

Multi-Path Transformer is Better: A Case Study on Neural Machine Translation

arXiv 2305.05948 ↗First posted 10 May 2023cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

For years the model performance in machine learning obeyed a power-law relationship with the model size. For the consideration of parameter efficiency, recent studies focus on increasing model depth rather than width to achieve better performance. In this paper, we study how model width affects the Transformer model through a parameter-efficient multi-path structure. To better fuse features extracted from different paths, we add three additional operations to each sublayer: a normalization at the end of each path, a cheap operation to produce more features, and a learnable weighted mechanism to fuse all features flexibly. Extensive experiments on 12 WMT machine translation tasks show that, with the same number of parameters, the shallower multi-path model can achieve similar or even better performance than the deeper model. It reveals that we should pay more attention to the multi-path structure, and there should be a balance between the model depth and width to train a better large-scale Transformer.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 31 of its 37 references that are in the map. First posted on arXiv 10 May 2023.

Builds on

31 of 37 references.

References · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  5. 2016Layer Normalizationcited by 5,669
  6. 2014Going Deeper with Convolutionscited by 4,241
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  8. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  9. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  10. 2017Mixed Precision Trainingcited by 1,156
  11. 2020On Layer Normalization in the Transformer Architecturecited by 922
  12. 2016Wide & Deep Learning for Recommender Systemscited by 747
19 more
  1. 2020ResNeSt: Split-Attention Networkscited by 445
  2. 2017The Expressive Power of Neural Networks: A View from the Widthcited by 320
  3. 2019Learning Deep Transformer Models for Machine Translationcited by 309
  4. 2020Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depthcited by 245
  5. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  6. 2020Understanding the Difficulty of Training Transformerscited by 210
  7. 2015The Power of Depth for Feedforward Neural Networkscited by 183
  8. 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 178
  9. 2019Transformers without Tears: Improving the Normalization of Self-Attentioncited by 153
  10. 2015Representation Benefits of Deep Feedforward Networkscited by 86
  11. 2021NormFormer: Improved Transformer Pretraining with Extra Normalizationcited by 83
  12. 2020Very Deep Transformers for Neural Machine Translationcited by 50
  13. 2017DiracNets: Training Very Deep Neural Networks Without Skip-Connectionscited by 47
  14. 2020Learning Light-Weight Translation Models from Deep Transformercited by 22
  15. 2017Weighted Transformer Network for Machine Translationcited by 16
  16. 2016A MultiPath Network for Object Detectioncited by 11
  17. 2020Multi-branch Attentive Transformercited by 10
  18. 2022Width is Less Important than Depth in ReLU Neural Networkscited by 10
  19. 2017Connectivity Learning in Multi-Branch Networkscited by 3

Built on by

Not cited yet.