# Joint Source-Target Self Attention with Locality Constraints

arXiv: https://arxiv.org/abs/1905.06596 · HTML: https://mlsanity.com/paper/1905.06596/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 May 2019.

## Built on by (most cited first, 4 of 4)

- [Examining Scaling and Transfer of Language Model Architectures for Machine Translation](https://mlsanity.com/paper/2202.00528/) (2022)
- [Paying More Attention to Self-attention: Improving Pre-trained Language Models via Attention Guiding](https://mlsanity.com/paper/2204.02922/) (2022)
- [IOT: Instance-wise Layer Reordering for Transformer Structures](https://mlsanity.com/paper/2103.03457/) (2021)
- [Investigating the translation capabilities of Large Language Models trained on parallel data only](https://mlsanity.com/paper/2406.09140/) (2024)

## Abstract (arXiv)

The dominant neural machine translation models are based on the encoder-decoder structure, and many of them rely on an unconstrained receptive field over source and target sequences. In this paper we study a new architecture that breaks with both conventions. Our simplified architecture consists in the decoder part of a transformer model, based on self-attention, but with locality constraints applied on the attention receptive field. As input for training, both source and target sentences are fed to the network, which is trained as a language model. At inference time, the target tokens are predicted autoregressively starting with the source sequence as previous tokens. The proposed model achieves a new state of the art of 35.7 BLEU on IWSLT'14 German-English and matches the best reported results in the literature on the WMT'14 English-German and WMT'14 English-French translation benchmarks.