# Neural Machine Translation with Supervised Attention

arXiv: https://arxiv.org/abs/1609.04186 · HTML: https://mlsanity.com/paper/1609.04186/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Sep 2016.

## Built on by (most cited first, 30 of 31)

- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Word Alignment by Fine-tuning Embeddings on Parallel Corpora](https://mlsanity.com/paper/2101.08231/) (2021)
- [Sequence-to-Sequence Learning with Latent Neural Grammars](https://mlsanity.com/paper/2109.01135/) (2021)
- [A Model-Agnostic Data Manipulation Method for Persona-based Dialogue Generation](https://mlsanity.com/paper/2204.09867/) (2022)
- [Do Context-Aware Translation Models Pay the Right Attention?](https://mlsanity.com/paper/2105.06977/) (2021)
- [Language Modeling, Lexical Translation, Reordering: The Training Process of NMT through the Lens of Classical SMT](https://mlsanity.com/paper/2109.01396/) (2021)
- [Modeling Dual Read/Write Paths for Simultaneous Machine Translation](https://mlsanity.com/paper/2203.09163/) (2022)
- [Modeling Concentrated Cross-Attention for Neural Machine Translation with Gaussian Mixture Model](https://mlsanity.com/paper/2109.05244/) (2021)
- [TranSmart: A Practical Interactive Machine Translation System](https://mlsanity.com/paper/2105.13072/) (2021)
- [Gaussian Multi-head Attention for Simultaneous Machine Translation](https://mlsanity.com/paper/2203.09072/) (2022)
- [Inducing and Using Alignments for Transition-based AMR Parsing](https://mlsanity.com/paper/2205.01464/) (2022)
- [DELTA: Pre-train a Discriminative Encoder for Legal Case Retrieval via Structural Word Alignment](https://mlsanity.com/paper/2403.18435/) (2024)
- [Discrete-continuous Action Space Policy Gradient-based Attention for Image-Text Matching](https://mlsanity.com/paper/2104.10406/) (2021)
- [Few-Shot Semantic Parsing for New Predicates](https://mlsanity.com/paper/2101.10708/) (2021)
- [Guiding Visual Question Answering with Attention Priors](https://mlsanity.com/paper/2205.12616/) (2022)
- [Hierarchical Phrase-based Sequence-to-Sequence Learning](https://mlsanity.com/paper/2211.07906/) (2022)
- [Enhanced Aspect-Based Sentiment Analysis Models with Progressive Self-supervised Attention Learning](https://mlsanity.com/paper/2103.03446/) (2021)
- [Abusive Language Detection in Heterogeneous Contexts: Dataset Collection and the Role of Supervised Attention](https://mlsanity.com/paper/2105.11119/) (2021)
- [Attention Forcing for Machine Translation](https://mlsanity.com/paper/2104.01264/) (2021)
- [Supervised Attention in Sequence-to-Sequence Models for Speech Recognition](https://mlsanity.com/paper/2204.12308/) (2022)
- [Word Alignment in the Era of Deep Learning: A Tutorial](https://mlsanity.com/paper/2212.00138/) (2022)
- [SemToken: Semantic-Aware Tokenization for Efficient Long-Context Language Modeling](https://mlsanity.com/paper/2508.15190/) (2025)
- [Supervised Visual Attention for Simultaneous Multimodal Machine Translation](https://mlsanity.com/paper/2201.09324/) (2022)
- [Unifying the Convergences in Multilingual Neural Machine Translation](https://mlsanity.com/paper/2205.01620/) (2022)
- [Metric-Type Identification for Multi-Level Header Numerical Tables in Scientific Papers](https://mlsanity.com/paper/2102.00819/) (2021)
- [An empirical analysis of phrase-based and neural machine translation](https://mlsanity.com/paper/2103.03108/) (2021)
- [Third-Party Aligner for Neural Word Alignments](https://mlsanity.com/paper/2211.04198/) (2022)
- [RCMHA: Relative Convolutional Multi-Head Attention for Natural Language Modelling](https://mlsanity.com/paper/2308.03429/) (2023)
- [Depth-induced Saliency Comparison Network for Diagnosis of Alzheimer's Disease via Jointly Analysis of Visual Stimuli and Eye Movements](https://mlsanity.com/paper/2403.10124/) (2024)
- [An Energy-based Model for Word-level AutoCompletion in Computer-aided Translation](https://mlsanity.com/paper/2407.20083/) (2024)

## Abstract (arXiv)

The attention mechanisim is appealing for neural machine translation, since it is able to dynam- ically encode a source sentence by generating a alignment between a target word and source words. Unfortunately, it has been proved to be worse than conventional alignment models in aligment accuracy. In this paper, we analyze and explain this issue from the point view of re- ordering, and propose a supervised attention which is learned with guidance from conventional alignment models. Experiments on two Chinese-to-English translation tasks show that the super- vised attention mechanism yields better alignments leading to substantial gains over the standard attention based NMT.