# On The Alignment Problem In Multi-Head Attention-Based Neural Machine Translation

arXiv: https://arxiv.org/abs/1809.03985 · HTML: https://mlsanity.com/paper/1809.03985/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Sep 2018.

## Built on by (most cited first, 17 of 17)

- [Word Alignment by Fine-tuning Embeddings on Parallel Corpora](https://mlsanity.com/paper/2101.08231/) (2021)
- [SALM: Speech-augmented Language Model with In-context Learning for Speech Recognition and Translation](https://mlsanity.com/paper/2310.09424/) (2023)
- [AligNART: Non-autoregressive Neural Machine Translation by Jointly Learning to Estimate Alignment and Translate](https://mlsanity.com/paper/2109.06481/) (2021)
- [Alignment Attention by Matching Key and Query Distributions](https://mlsanity.com/paper/2110.12567/) (2021)
- [A study of latent monotonic attention variants](https://mlsanity.com/paper/2103.16710/) (2021)
- [Is Encoder-Decoder Redundant for Neural Machine Translation?](https://mlsanity.com/paper/2210.11807/) (2022)
- [MirrorAlign: A Super Lightweight Unsupervised Word Alignment Model via Cross-Lingual Contrastive Learning](https://mlsanity.com/paper/2102.04009/) (2021)
- [DICTDIS: Dictionary Constrained Disambiguation for Improved NMT](https://mlsanity.com/paper/2210.06996/) (2022)
- [UDAAN: Machine Learning based Post-Editing tool for Document Translation](https://mlsanity.com/paper/2203.01644/) (2022)
- [Neighbors Are Not Strangers: Improving Non-Autoregressive Translation under Low-Frequency Lexical Constraints](https://mlsanity.com/paper/2204.13355/) (2022)
- [Word Alignment in the Era of Deep Learning: A Tutorial](https://mlsanity.com/paper/2212.00138/) (2022)
- [Does mBERT understand Romansh? Evaluating word embeddings using word alignment](https://mlsanity.com/paper/2306.08702/) (2023)
- [Emergent Communication with Attention](https://mlsanity.com/paper/2305.10920/) (2023)
- [Accurate Online Posterior Alignments for Principled Lexically-Constrained Decoding](https://mlsanity.com/paper/2204.00871/) (2022)
- [Predicting Human Translation Difficulty with Neural Machine Translation](https://mlsanity.com/paper/2312.11852/) (2023)
- [Fine Grained Human Evaluation for English-to-Chinese Machine Translation: A Case Study on Scientific Text](https://mlsanity.com/paper/2110.14766/) (2021)
- [SilverAlign: MT-Based Silver Data Algorithm For Evaluating Word Alignment](https://mlsanity.com/paper/2210.06207/) (2022)

## Abstract (arXiv)

This work investigates the alignment problem in state-of-the-art multi-head attention models based on the transformer architecture. We demonstrate that alignment extraction in transformer models can be improved by augmenting an additional alignment head to the multi-head source-to-target attention component. This is used to compute sharper attention weights. We describe how to use the alignment head to achieve competitive performance. To study the effect of adding the alignment head, we simulate a dictionary-guided translation task, where the user wants to guide translation using pre-defined dictionary entries. Using the proposed approach, we achieve up to 3.8 % BLEU improvement when using the dictionary, in comparison to 2.4 % BLEU in the baseline case. We also propose alignment pruning to speed up decoding in alignment-based neural machine translation (ANMT), which speeds up translation by a factor of 1.8 without loss in translation performance. We carry out experiments on the shared WMT 2016 English→Romanian news task and the BOLT Chinese→English discussion forum task.