# Weighted Transformer Network for Machine Translation

arXiv: https://arxiv.org/abs/1711.02132 · HTML: https://mlsanity.com/paper/1711.02132/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Nov 2017.

## Built on by (most cited first, 16 of 16)

- [Referred by Multi-Modality: A Unified Temporal Transformer for Video Object Segmentation](https://mlsanity.com/paper/2305.16318/) (2023)
- [Attention, please! A survey of Neural Attention Models in Deep Learning](https://mlsanity.com/paper/2103.16775/) (2021)
- [Mask Attention Networks: Rethinking and Strengthen Transformer](https://mlsanity.com/paper/2103.13597/) (2021)
- [UniDrop: A Simple yet Effective Technique to Improve Transformer without Extra Cost](https://mlsanity.com/paper/2104.04946/) (2021)
- [Improving Transformers with Dynamically Composable Multi-Head Attention](https://mlsanity.com/paper/2405.08553/) (2024)
- [IOT: Instance-wise Layer Reordering for Transformer Structures](https://mlsanity.com/paper/2103.03457/) (2021)
- [Devil is in the Uniformity: Exploring Diverse Learners within Transformer for Image Restoration](https://mlsanity.com/paper/2503.20174/) (2025)
- [TranSFormer: Slow-Fast Transformer for Machine Translation](https://mlsanity.com/paper/2305.16982/) (2023)
- [Understanding Parameter Sharing in Transformers](https://mlsanity.com/paper/2306.09380/) (2023)
- [Scalable Transformers for Neural Machine Translation](https://mlsanity.com/paper/2106.02242/) (2021)
- [Finding the Pillars of Strength for Multi-Head Attention](https://mlsanity.com/paper/2305.14380/) (2023)
- [PartialFormer: Modeling Part Instead of Whole for Machine Translation](https://mlsanity.com/paper/2310.14921/) (2023)
- [Leveraging Pre-trained Models for Failure Analysis Triplets Generation](https://mlsanity.com/paper/2210.17497/) (2022)
- [Program Generation from Diverse Video Demonstrations](https://mlsanity.com/paper/2302.00178/) (2023)
- [Multi-Path Transformer is Better: A Case Study on Neural Machine Translation](https://mlsanity.com/paper/2305.05948/) (2023)
- [Unveiling the frontiers of deep learning: innovations shaping diverse domains](https://mlsanity.com/paper/2309.02712/) (2023)

## Abstract (arXiv)

State-of-the-art results on neural machine translation often use attentional sequence-to-sequence models with some form of convolution or recursion. Vaswani et al. (2017) propose a new architecture that avoids recurrence and convolution completely. Instead, it uses only self-attention and feed-forward layers. While the proposed architecture achieves state-of-the-art results on several machine translation tasks, it requires a large number of parameters and training iterations to converge. We propose Weighted Transformer, a Transformer with modified attention layers, that not only outperforms the baseline network in BLEU score but also converges 15-40% faster. Specifically, we replace the multi-head attention by multiple self-attention branches that the model learns to combine during the training process. Our model improves the state-of-the-art performance by 0.5 BLEU points on the WMT 2014 English-to-German translation task and by 0.4 on the English-to-French translation task.