# Why Self-Attention? A Targeted Evaluation of Neural Machine Translation Architectures

arXiv: https://arxiv.org/abs/1808.08946 · HTML: https://mlsanity.com/paper/1808.08946/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Aug 2018.

## Built on by (most cited first, 30 of 30)

- [Molformer: Motif-based Transformer on 3D Heterogeneous Molecular Graphs](https://mlsanity.com/paper/2110.01191/) (2021)
- [Applying QNLP to sentiment analysis in finance](https://mlsanity.com/paper/2307.11788/) (2023)
- [Evolving Attention with Residual Convolutions](https://mlsanity.com/paper/2102.12895/) (2021)
- [TraverseNet: Unifying Space and Time in Message Passing for Traffic Forecasting](https://mlsanity.com/paper/2109.02474/) (2021)
- [CAT-Net: A Cross-Slice Attention Transformer Model for Prostate Zonal Segmentation in MRI](https://mlsanity.com/paper/2203.15163/) (2022)
- [CaEGCN: Cross-Attention Fusion based Enhanced Graph Convolutional Network for Clustering](https://mlsanity.com/paper/2101.06883/) (2021)
- [Contrastive Learning for Context-aware Neural Machine TranslationUsing Coreference Information](https://mlsanity.com/paper/2109.05712/) (2021)
- [Evaluating Document Coherence Modelling](https://mlsanity.com/paper/2103.10133/) (2021)
- [Breaking Down Multilingual Machine Translation](https://mlsanity.com/paper/2110.08130/) (2021)
- [NeurIPS 2020 NLC2CMD Competition: Translating Natural Language to Bash Commands](https://mlsanity.com/paper/2103.02523/) (2021)
- [Revisiting Negation in Neural Machine Translation](https://mlsanity.com/paper/2107.12203/) (2021)
- [MRTNet: Multi-Resolution Temporal Network for Video Sentence Grounding](https://mlsanity.com/paper/2212.13163/) (2022)
- [PENELOPIE: Enabling Open Information Extraction for the Greek Language through Machine Translation](https://mlsanity.com/paper/2103.15075/) (2021)
- [Convolution-enhanced Evolving Attention Networks](https://mlsanity.com/paper/2212.08330/) (2022)
- [Semantic-guided Automatic Natural Image Matting with Trimap Generation Network and Light-weight Non-local Attention](https://mlsanity.com/paper/2103.17020/) (2021)
- [Blind Source Separation of Single-Channel Mixtures via Multi-Encoder Autoencoders](https://mlsanity.com/paper/2309.07138/) (2023)
- [Foundations of Multisensory Artificial Intelligence](https://mlsanity.com/paper/2404.18976/) (2024)
- [Interpretable Emergent Language Using Inter-Agent Transformers](https://mlsanity.com/paper/2505.02215/) (2025)
- [Temporal State Transport in Video Generation: Diagnosing and Correcting Spectral Imbalance](https://mlsanity.com/paper/2609.08505/) (2026)
- [An empirical analysis of phrase-based and neural machine translation](https://mlsanity.com/paper/2103.03108/) (2021)
- [TransfoRNN: Capturing the Sequential Information in Self-Attention Representations for Language Modeling](https://mlsanity.com/paper/2104.01572/) (2021)
- [Cascaded Head-colliding Attention](https://mlsanity.com/paper/2105.14850/) (2021)
- [Fine Grained Human Evaluation for English-to-Chinese Machine Translation: A Case Study on Scientific Text](https://mlsanity.com/paper/2110.14766/) (2021)
- [Is the Computation of Abstract Sameness Relations Human-Like in Neural Language Models?](https://mlsanity.com/paper/2205.06149/) (2022)
- [CoCoFormer: A controllable feature-rich polyphonic music generation method](https://mlsanity.com/paper/2310.09843/) (2023)
- [Hashing it Out: Predicting Unhealthy Conversations on Twitter](https://mlsanity.com/paper/2311.10596/) (2023)
- [Self-attention-based non-linear basis transformations for compact latent space modelling of dynamic optical fibre transmission matrices](https://mlsanity.com/paper/2406.07775/) (2024)
- [Mention Attention for Pronoun Translation](https://mlsanity.com/paper/2412.14829/) (2024)
- [Automated Invoice Data Extraction: Using LLM and OCR](https://mlsanity.com/paper/2511.05547/) (2025)
- [Communicability-Inspired Positional Encoding (CIPE)](https://mlsanity.com/paper/2606.25293/) (2026)

## Abstract (arXiv)

Recently, non-recurrent architectures (convolutional, self-attentional) have outperformed RNNs in neural machine translation. CNNs and self-attentional networks can connect distant words via shorter network paths than RNNs, and it has been speculated that this improves their ability to model long-range dependencies. However, this theoretical argument has not been tested empirically, nor have alternative explanations for their strong performance been explored in-depth. We hypothesize that the strong performance of CNNs and self-attentional networks could also be due to their ability to extract semantic features from the source text, and we evaluate RNNs, CNNs and self-attention networks on two tasks: subject-verb agreement (where capturing long-range dependencies is required) and word sense disambiguation (where semantic feature extraction is required). Our experimental results show that: 1) self-attentional networks and CNNs do not outperform RNNs in modeling subject-verb agreement over long distances; 2) self-attentional networks perform distinctly better than RNNs and CNNs on word sense disambiguation.