Heterogeneous Encoders Scaling In The Transformer For Neural Machine Translation
Not on the map. Why?
Abstract (arXiv)
Although the Transformer is currently the best-performing architecture in the homogeneous configuration (self-attention only) in Neural Machine Translation, many State-of-the-Art models in Natural Language Processing are made of a combination of different Deep Learning approaches. However, these models often focus on combining a couple of techniques only and it is unclear why some methods are chosen over others. In this work, we investigate the effectiveness of integrating an increasing number of heterogeneous methods. Based on a simple combination strategy and performance-driven synergy criteria, we designed the Multi-Encoder Transformer, which consists of up to five diverse encoders. Results showcased that our approach can improve the quality of the translation across a variety of languages and dataset sizes and it is particularly effective in low-resource languages where we observed a maximum increase of 7.16 BLEU compared to the single-encoder model.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 33 of its 40 references that are in the map. First posted on arXiv 26 Dec 2023.
Builds on
33 of 40 references.
References · 33
- 2017Attention Is All You Needcited by 60,361
- 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
- 2020Longformer: The Long-Document Transformercited by 3,413
- 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
- 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
- 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
- 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
- 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
- 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,026
- 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 569
- 2017Convolutional Sequence to Sequence Learningcited by 552
- 2021FNet: Mixing Tokens with Fourier Transformscited by 347
21 more
- 2019Attention on Attention for Image Captioningcited by 289
- 2019Pay Less Attention with Lightweight and Dynamic Convolutionscited by 211
- 2018Marian: Fast Neural Machine Translation in C++cited by 203
- 2019BPE-Dropout: Simple and Effective Subword Regularizationcited by 195
- 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 178
- 2018QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehensioncited by 164
- 2019Transformers without Tears: Improving the Normalization of Self-Attentioncited by 153
- 2018The Best of Both Worlds: Combining Recent Advances in Neural Machine Translationcited by 102
- 2020Low-resource Languages: A Review of Past Work and Future Challengescited by 91
- 2016Quasi-Recurrent Neural Networkscited by 87
- 2021Sparse is Enough in Scaling Transformerscited by 86
- 2018Training Tips for the Transformer Modelcited by 85
- 2022Squeezeformer: An Efficient Transformer for Automatic Speech Recognitioncited by 51
- 2020Very Deep Transformers for Neural Machine Translationcited by 50
- 2019Very Deep Self-Attention Networks for End-to-End Speech Recognitioncited by 46
- 2020ConvTransformer: A Convolutional Transformer Network for Video Frame Synthesiscited by 34
- 2019Convolutional Self-Attention Networkscited by 32
- 2019Modeling Recurrence for Transformercited by 26
- 2018Input Combination Strategies for Multi-Source Transformer Decodercited by 22
- 2022Exploiting Multiple Sequence Lengths in Fast End to End Training for Image Captioningcited by 18
- 2022Exploring the sequence length bottleneck in the Transformer for Image Captioningcited by 2
Built on by
Citing papers per year: 2024: 1.
1 paper.