# Adding Interpretable Attention to Neural Translation Models Improves Word Alignment

arXiv: https://arxiv.org/abs/1901.11359 · HTML: https://mlsanity.com/paper/1901.11359/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Jan 2019.

## Built on by (most cited first, 30 of 30)

- [Word Alignment by Fine-tuning Embeddings on Parallel Corpora](https://mlsanity.com/paper/2101.08231/) (2021)
- [Attention, please! A survey of Neural Attention Models in Deep Learning](https://mlsanity.com/paper/2103.16775/) (2021)
- [Towards Opening the Black Box of Neural Machine Translation: Source and Target Interpretations of the Transformer](https://mlsanity.com/paper/2205.11631/) (2022)
- [Explaining How Transformers Use Context to Build Predictions](https://mlsanity.com/paper/2305.12535/) (2023)
- [Sparse Attention with Linear Units](https://mlsanity.com/paper/2104.07012/) (2021)
- [Attention as a Guide for Simultaneous Speech Translation](https://mlsanity.com/paper/2212.07850/) (2022)
- [Toxicity in Multilingual Machine Translation at Scale](https://mlsanity.com/paper/2210.03070/) (2022)
- [Bilingual Lexicon Induction via Unsupervised Bitext Construction and Word Alignment](https://mlsanity.com/paper/2101.00148/) (2021)
- [StreamAtt: Direct Streaming Speech-to-Text Translation with Attention-based Audio History Selection](https://mlsanity.com/paper/2406.06097/) (2024)
- [On-the-Fly Attention Modulation for Neural Generation](https://mlsanity.com/paper/2101.00371/) (2021)
- [Attention Weights in Transformer NMT Fail Aligning Words Between Sequences but Largely Explain Model Predictions](https://mlsanity.com/paper/2109.05853/) (2021)
- [Attention Consistency for LLMs Explanation](https://mlsanity.com/paper/2509.17178/) (2025)
- [WSPAlign: Word Alignment Pre-training via Large-Scale Weakly Supervised Span Prediction](https://mlsanity.com/paper/2306.05644/) (2023)
- [DELTA: Pre-train a Discriminative Encoder for Legal Case Retrieval via Structural Word Alignment](https://mlsanity.com/paper/2403.18435/) (2024)
- [Multilingual Sentence Transformer as A Multilingual Word Aligner](https://mlsanity.com/paper/2301.12140/) (2023)
- [MirrorAlign: A Super Lightweight Unsupervised Word Alignment Model via Cross-Lingual Contrastive Learning](https://mlsanity.com/paper/2102.04009/) (2021)
- [Checks and Strategies for Enabling Code-Switched Machine Translation](https://mlsanity.com/paper/2210.05096/) (2022)
- [Gloss Alignment Using Word Embeddings](https://mlsanity.com/paper/2308.04248/) (2023)
- [Cross-Align: Modeling Deep Cross-lingual Interactions for Word Alignment](https://mlsanity.com/paper/2210.04141/) (2022)
- [Accurate Online Posterior Alignments for Principled Lexically-Constrained Decoding](https://mlsanity.com/paper/2204.00871/) (2022)
- [Automatic Correction of Human Translations](https://mlsanity.com/paper/2206.08593/) (2022)
- [SBAAM! Eliminating Transcript Dependency in Automatic Subtitling](https://mlsanity.com/paper/2405.10741/) (2024)
- [Towards Global AI Inclusivity: A Large-Scale Multilingual Terminology Dataset (GIST)](https://mlsanity.com/paper/2412.18367/) (2024)
- [Leveraging Neural Machine Translation for Word Alignment](https://mlsanity.com/paper/2103.17250/) (2021)
- [Embedding-Enhanced Giza++: Improving Alignment in Low- and High- Resource Scenarios Using Embedding Space Geometry](https://mlsanity.com/paper/2104.08721/) (2021)
- [BERT Cannot Align Characters](https://mlsanity.com/paper/2109.09700/) (2021)
- [SilverAlign: MT-Based Silver Data Algorithm For Evaluating Word Alignment](https://mlsanity.com/paper/2210.06207/) (2022)
- [Third-Party Aligner for Neural Word Alignments](https://mlsanity.com/paper/2211.04198/) (2022)
- [Cross-Lingual Word Alignment for ASEAN Languages with Contrastive Learning](https://mlsanity.com/paper/2407.05054/) (2024)
- [Evaluating Explainable AI Attribution Methods in Neural Machine Translation via Attention-Guided Knowledge Distillation](https://mlsanity.com/paper/2603.11342/) (2026)

## Abstract (arXiv)

Multi-layer models with multiple attention heads per layer provide superior translation quality compared to simpler and shallower models, but determining what source context is most relevant to each target word is more challenging as a result. Therefore, deriving high-accuracy word alignments from the activations of a state-of-the-art neural machine translation model is an open challenge. We propose a simple model extension to the Transformer architecture that makes use of its hidden representations and is restricted to attend solely on encoder information to predict the next word. It can be trained on bilingual data without word-alignment information. We further introduce a novel alignment inference procedure which applies stochastic gradient descent to directly optimize the attention activations towards a given target word. The resulting alignments dramatically outperform the naive approach to interpreting Transformer attention activations, and are comparable to Giza++ on two publicly available data sets.