# Effective Approaches to Attention-based Neural Machine Translation

arXiv: https://arxiv.org/abs/1508.04025 · HTML: https://mlsanity.com/paper/1508.04025/

## Key facts

- Cited by 1,026 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Aug 2015.

## Built on by (most cited first, 30 of 1026)

- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [Are NLP Models really able to Solve Simple Math Word Problems?](https://mlsanity.com/paper/2103.07191/) (2021)
- [How Attentive are Graph Attention Networks?](https://mlsanity.com/paper/2105.14491/) (2021)
- [ProgPrompt: Generating Situated Robot Task Plans using Large Language Models](https://mlsanity.com/paper/2209.11302/) (2022)
- [Dynabench: Rethinking Benchmarking in NLP](https://mlsanity.com/paper/2104.14337/) (2021)
- [Unified Pre-training for Program Understanding and Generation](https://mlsanity.com/paper/2103.06333/) (2021)
- [ScanQA: 3D Question Answering for Spatial Scene Understanding](https://mlsanity.com/paper/2112.10482/) (2021)
- [AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting](https://mlsanity.com/paper/2103.14023/) (2021)
- [Transformer for Partial Differential Equations' Operator Learning](https://mlsanity.com/paper/2205.13671/) (2022)
- [Escaping the Big Data Paradigm with Compact Transformers](https://mlsanity.com/paper/2104.05704/) (2021)
- [Training Spiking Neural Networks Using Lessons From Deep Learning](https://mlsanity.com/paper/2109.12894/) (2021)
- [A Contrastive Framework for Neural Text Generation](https://mlsanity.com/paper/2202.06417/) (2022)
- [Word Alignment by Fine-tuning Embeddings on Parallel Corpora](https://mlsanity.com/paper/2101.08231/) (2021)
- [Thinking Like Transformers](https://mlsanity.com/paper/2106.06981/) (2021)
- [TorchMD-NET: Equivariant Transformers for Neural Network based Molecular Potentials](https://mlsanity.com/paper/2202.02541/) (2022)
- [Improving Sign Language Translation with Monolingual Data by Sign Back-Translation](https://mlsanity.com/paper/2105.12397/) (2021)
- [Mega: Moving Average Equipped Gated Attention](https://mlsanity.com/paper/2209.10655/) (2022)
- [Implicit Chain of Thought Reasoning via Knowledge Distillation](https://mlsanity.com/paper/2311.01460/) (2023)
- [Retrieval Augmented Code Generation and Summarization](https://mlsanity.com/paper/2108.11601/) (2021)
- [Inductive Biases and Variable Creation in Self-Attention Mechanisms](https://mlsanity.com/paper/2110.10090/) (2021)
- [Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Maps](https://mlsanity.com/paper/2407.07071/) (2024)
- [A Unified Generative Framework for Various NER Subtasks](https://mlsanity.com/paper/2106.01223/) (2021)
- [Scalable Transformer for PDE Surrogate Modeling](https://mlsanity.com/paper/2305.17560/) (2023)
- [PyTorch Geometric Temporal: Spatiotemporal Signal Processing with Neural Machine Learning Models](https://mlsanity.com/paper/2104.07788/) (2021)
- [DanceFormer: Music Conditioned 3D Dance Generation with Parametric Motion Transformer](https://mlsanity.com/paper/2103.10206/) (2021)
- [Gloss-free Sign Language Translation: Improving from Visual-Language Pretraining](https://mlsanity.com/paper/2307.14768/) (2023)
- [Recent advances and clinical applications of deep learning in medical image analysis](https://mlsanity.com/paper/2105.13381/) (2021)
- [How to Find Your Friendly Neighborhood: Graph Attention Design with Self-Supervision](https://mlsanity.com/paper/2204.04879/) (2022)

## Abstract (arXiv)

An attentional mechanism has lately been used to improve neural machine translation (NMT) by selectively focusing on parts of the source sentence during translation. However, there has been little work exploring useful architectures for attention-based NMT. This paper examines two simple and effective classes of attentional mechanism: a global approach which always attends to all source words and a local one that only looks at a subset of source words at a time. We demonstrate the effectiveness of both approaches over the WMT translation tasks between English and German in both directions. With local attention, we achieve a significant gain of 5.0 BLEU points over non-attentional systems which already incorporate known techniques such as dropout. Our ensemble model using different attention architectures has established a new state-of-the-art result in the WMT'15 English to German translation task with 25.9 BLEU points, an improvement of 1.0 BLEU points over the existing best system backed by NMT and an n-gram reranker.