mlsanityData to 7 Oct 2026

What does Attention in Neural Machine Translation Pay Attention to?

arXiv 1710.03348 ↗First posted 9 Oct 2017cs.CLcited by 30

Not on the map. Why?

Abstract (arXiv)

Attention in neural machine translation provides the possibility to encode relevant parts of the source sentence at each translation step. As a result, attention is considered to be an alignment model as well. However, there is no work that specifically studies attention and provides analysis of what is being learned by attention models. Thus, the question still remains that how attention is similar or different from the traditional alignment. In this paper, we provide detailed analysis of attention and compare it to traditional alignment. We answer the question of whether attention is only capable of modelling translational equivalent or it captures more information. We show that attention is different from alignment in some cases and is capturing useful information other than alignments.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). First posted on arXiv 9 Oct 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 10; 2022: 5; 2023: 4; 2024: 5; 2025: 4; 2026: 2.

30 papers.

Sort:

By area:

  1. 2023Training-free Diffusion Model Adaptation for Variable-Sized Text-to-Image Synthesiscited by 55
  2. 2022Entropy-based Attention Regularization Frees Unintended Bias Mitigation from Listscited by 54
  3. 2021Sparse Attention with Linear Unitscited by 41
  4. 2021Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignmentcited by 41
  5. 2024Loss Landscape Degeneracy and Stagewise Development in Transformerscited by 39
  6. 2022STNDT: Modeling Neural Population Activity with a Spatiotemporal Transformercited by 23
  7. 2021Language Modeling, Lexical Translation, Reordering: The Training Process of NMT through the Lens of Classical SMTcited by 21
  8. 2022Modeling Dual Read/Write Paths for Simultaneous Machine Translationcited by 21
  9. 2021Interpreting Deep Learning Models in Natural Language Processing: A Reviewcited by 19
  10. 2021Modeling Concentrated Cross-Attention for Neural Machine Translation with Gaussian Mixture Modelcited by 18
  11. 2023Linear Log-Normal Attention with Unbiased Concentrationcited by 11
  12. 2021What do End-to-End Speech Models Learn about Speaker, Language and Channel Information? A Layer-wise and Neuron-level Analysiscited by 9
18 more
  1. 2021Enhanced Aspect-Based Sentiment Analysis Models with Progressive Self-supervised Attention Learningcited by 5
  2. 2021MirrorAlign: A Super Lightweight Unsupervised Word Alignment Model via Cross-Lingual Contrastive Learningcited by 4
  3. 2022Sparse Attentive Memory Network for Click-through Rate Prediction with Long Sequencescited by 3
  4. 2022Word Alignment in the Era of Deep Learning: A Tutorialcited by 3
  5. 2024ReLU's Revival: On the Entropic Overload in Normalization-Free Large Language Modelscited by 3
  6. 2024AERO: Entropy-Guided Framework for Private LLM Inferencecited by 3
  7. 2021Understanding How Encoder-Decoder Architectures Attendcited by 2
  8. 2023Robustifying Point Cloud Networks by Refocusingcited by 2
  9. 2023SpeechAlign: a Framework for Speech Translation Alignment Evaluationcited by 2
  10. 2024Cross-lingual Contextualized Phrase Retrievalcited by 1
  11. 2024OTTAWA: Optimal TransporT Adaptive Word Aligner for Hallucination and Omission Translation Errors Detectioncited by 1
  12. 2025MirrorShield: Towards Universal Defense Against Jailbreaks via Entropy-Guided Mirror Craftingcited by 1
  13. 2025On the Role of Hidden States of Modern Hopfield Network in Transformercited by 1
  14. 2021An empirical analysis of phrase-based and neural machine translationcited by 0
  15. 2025Deep Learning Model Deployment in Multiple Cloud Providers: an Exploratory Study Using Low Computing Power Environmentscited by 0
  16. 2025TransAlign: Machine Translation Encoders are Strong Word Aligners, Toocited by 0
  17. 2026Evaluating Explainable AI Attribution Methods in Neural Machine Translation via Attention-Guided Knowledge Distillationcited by 0
  18. 2026Systematic Analysis of Large Language Models and Transformer-Based Machine Translation for English-Tamil and Tamil-English Across Diverse Datasetscited by 0