# What does Attention in Neural Machine Translation Pay Attention to?

arXiv: https://arxiv.org/abs/1710.03348 · HTML: https://mlsanity.com/paper/1710.03348/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Oct 2017.

## Built on by (most cited first, 30 of 30)

- [Training-free Diffusion Model Adaptation for Variable-Sized Text-to-Image Synthesis](https://mlsanity.com/paper/2306.08645/) (2023)
- [Entropy-based Attention Regularization Frees Unintended Bias Mitigation from Lists](https://mlsanity.com/paper/2203.09192/) (2022)
- [Sparse Attention with Linear Units](https://mlsanity.com/paper/2104.07012/) (2021)
- [Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment](https://mlsanity.com/paper/2106.06381/) (2021)
- [Loss Landscape Degeneracy and Stagewise Development in Transformers](https://mlsanity.com/paper/2402.02364/) (2024)
- [STNDT: Modeling Neural Population Activity with a Spatiotemporal Transformer](https://mlsanity.com/paper/2206.04727/) (2022)
- [Language Modeling, Lexical Translation, Reordering: The Training Process of NMT through the Lens of Classical SMT](https://mlsanity.com/paper/2109.01396/) (2021)
- [Modeling Dual Read/Write Paths for Simultaneous Machine Translation](https://mlsanity.com/paper/2203.09163/) (2022)
- [Interpreting Deep Learning Models in Natural Language Processing: A Review](https://mlsanity.com/paper/2110.10470/) (2021)
- [Modeling Concentrated Cross-Attention for Neural Machine Translation with Gaussian Mixture Model](https://mlsanity.com/paper/2109.05244/) (2021)
- [Linear Log-Normal Attention with Unbiased Concentration](https://mlsanity.com/paper/2311.13541/) (2023)
- [What do End-to-End Speech Models Learn about Speaker, Language and Channel Information? A Layer-wise and Neuron-level Analysis](https://mlsanity.com/paper/2107.00439/) (2021)
- [Enhanced Aspect-Based Sentiment Analysis Models with Progressive Self-supervised Attention Learning](https://mlsanity.com/paper/2103.03446/) (2021)
- [MirrorAlign: A Super Lightweight Unsupervised Word Alignment Model via Cross-Lingual Contrastive Learning](https://mlsanity.com/paper/2102.04009/) (2021)
- [Sparse Attentive Memory Network for Click-through Rate Prediction with Long Sequences](https://mlsanity.com/paper/2208.04022/) (2022)
- [Word Alignment in the Era of Deep Learning: A Tutorial](https://mlsanity.com/paper/2212.00138/) (2022)
- [ReLU's Revival: On the Entropic Overload in Normalization-Free Large Language Models](https://mlsanity.com/paper/2410.09637/) (2024)
- [AERO: Entropy-Guided Framework for Private LLM Inference](https://mlsanity.com/paper/2410.13060/) (2024)
- [Understanding How Encoder-Decoder Architectures Attend](https://mlsanity.com/paper/2110.15253/) (2021)
- [Robustifying Point Cloud Networks by Refocusing](https://mlsanity.com/paper/2308.05525/) (2023)
- [SpeechAlign: a Framework for Speech Translation Alignment Evaluation](https://mlsanity.com/paper/2309.11585/) (2023)
- [Cross-lingual Contextualized Phrase Retrieval](https://mlsanity.com/paper/2403.16820/) (2024)
- [OTTAWA: Optimal TransporT Adaptive Word Aligner for Hallucination and Omission Translation Errors Detection](https://mlsanity.com/paper/2406.01919/) (2024)
- [MirrorShield: Towards Universal Defense Against Jailbreaks via Entropy-Guided Mirror Crafting](https://mlsanity.com/paper/2503.12931/) (2025)
- [On the Role of Hidden States of Modern Hopfield Network in Transformer](https://mlsanity.com/paper/2511.20698/) (2025)
- [An empirical analysis of phrase-based and neural machine translation](https://mlsanity.com/paper/2103.03108/) (2021)
- [Deep Learning Model Deployment in Multiple Cloud Providers: an Exploratory Study Using Low Computing Power Environments](https://mlsanity.com/paper/2503.23988/) (2025)
- [TransAlign: Machine Translation Encoders are Strong Word Aligners, Too](https://mlsanity.com/paper/2510.27337/) (2025)
- [Evaluating Explainable AI Attribution Methods in Neural Machine Translation via Attention-Guided Knowledge Distillation](https://mlsanity.com/paper/2603.11342/) (2026)
- [Systematic Analysis of Large Language Models and Transformer-Based Machine Translation for English-Tamil and Tamil-English Across Diverse Datasets](https://mlsanity.com/paper/2607.24515/) (2026)

## Abstract (arXiv)

Attention in neural machine translation provides the possibility to encode relevant parts of the source sentence at each translation step. As a result, attention is considered to be an alignment model as well. However, there is no work that specifically studies attention and provides analysis of what is being learned by attention models. Thus, the question still remains that how attention is similar or different from the traditional alignment. In this paper, we provide detailed analysis of attention and compare it to traditional alignment. We answer the question of whether attention is only capable of modelling translational equivalent or it captures more information. We show that attention is different from alignment in some cases and is capturing useful information other than alignments.