# Accurate Word Alignment Induction from Neural Machine Translation

arXiv: https://arxiv.org/abs/2004.14837 · HTML: https://mlsanity.com/paper/2004.14837/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Apr 2020.

## Built on by (most cited first, 30 of 32)

- [A Primer on Pretrained Multilingual Language Models](https://mlsanity.com/paper/2107.00676/) (2021)
- [Towards Opening the Black Box of Neural Machine Translation: Source and Target Interpretations of the Transformer](https://mlsanity.com/paper/2205.11631/) (2022)
- [Sparse Attention with Linear Units](https://mlsanity.com/paper/2104.07012/) (2021)
- [Information-Transport-based Policy for Simultaneous Translation](https://mlsanity.com/paper/2210.12357/) (2022)
- [AlignAtt: Using Attention-based Audio-Translation Alignments as a Guide for Simultaneous Speech Translation](https://mlsanity.com/paper/2305.11408/) (2023)
- [Attention as a Guide for Simultaneous Speech Translation](https://mlsanity.com/paper/2212.07850/) (2022)
- [Toxicity in Multilingual Machine Translation at Scale](https://mlsanity.com/paper/2210.03070/) (2022)
- [Language Modeling, Lexical Translation, Reordering: The Training Process of NMT through the Lens of Classical SMT](https://mlsanity.com/paper/2109.01396/) (2021)
- [StreamAtt: Direct Streaming Speech-to-Text Translation with Attention-based Audio History Selection](https://mlsanity.com/paper/2406.06097/) (2024)
- [TranSmart: A Practical Interactive Machine Translation System](https://mlsanity.com/paper/2105.13072/) (2021)
- [Translate & Fill: Improving Zero-Shot Multilingual Semantic Parsing with Synthetic Data](https://mlsanity.com/paper/2109.04319/) (2021)
- [Salute the Classic: Revisiting Challenges of Machine Translation in the Age of Large Language Models](https://mlsanity.com/paper/2401.08350/) (2024)
- [Attention Weights in Transformer NMT Fail Aligning Words Between Sequences but Largely Explain Model Predictions](https://mlsanity.com/paper/2109.05853/) (2021)
- [Integrating Vectorized Lexical Constraints for Neural Machine Translation](https://mlsanity.com/paper/2203.12210/) (2022)
- [Multilingual Sentence Transformer as A Multilingual Word Aligner](https://mlsanity.com/paper/2301.12140/) (2023)
- [Generative Retrieval as Multi-Vector Dense Retrieval](https://mlsanity.com/paper/2404.00684/) (2024)
- [Graph Neural Networks for Multiparallel Word Alignment](https://mlsanity.com/paper/2203.08654/) (2022)
- [Meeting the Needs of Low-Resource Languages: The Value of Automatic Alignments via Pretrained Models](https://mlsanity.com/paper/2302.07912/) (2023)
- [How Effective is Byte Pair Encoding for Out-Of-Vocabulary Words in Neural Machine Translation?](https://mlsanity.com/paper/2208.05225/) (2022)
- [Checks and Strategies for Enabling Code-Switched Machine Translation](https://mlsanity.com/paper/2210.05096/) (2022)
- [Word Alignment in the Era of Deep Learning: A Tutorial](https://mlsanity.com/paper/2212.00138/) (2022)
- [BinaryAlign: Word Alignment as Binary Sequence Labeling](https://mlsanity.com/paper/2407.12881/) (2024)
- [Cross-Align: Modeling Deep Cross-lingual Interactions for Word Alignment](https://mlsanity.com/paper/2210.04141/) (2022)
- [Categorizing Semantic Representations for Neural Machine Translation](https://mlsanity.com/paper/2210.06709/) (2022)
- [Accurate Online Posterior Alignments for Principled Lexically-Constrained Decoding](https://mlsanity.com/paper/2204.00871/) (2022)
- [Noisy Parallel Data Alignment](https://mlsanity.com/paper/2301.09685/) (2023)
- [SBAAM! Eliminating Transcript Dependency in Automatic Subtitling](https://mlsanity.com/paper/2405.10741/) (2024)
- [Towards Global AI Inclusivity: A Large-Scale Multilingual Terminology Dataset (GIST)](https://mlsanity.com/paper/2412.18367/) (2024)
- [Cross-Attention is Half Explanation in Speech-to-Text Models](https://mlsanity.com/paper/2509.18010/) (2025)
- [Leveraging Neural Machine Translation for Word Alignment](https://mlsanity.com/paper/2103.17250/) (2021)

## Abstract (arXiv)

Despite its original goal to jointly learn to align and translate, prior researches suggest that Transformer captures poor word alignments through its attention mechanism. In this paper, we show that attention weights DO capture accurate word alignments and propose two novel word alignment induction methods Shift-Att and Shift-AET. The main idea is to induce alignments at the step when the to-be-aligned target token is the decoder input rather than the decoder output as in previous work. Shift-Att is an interpretation method that induces alignments from the attention weights of Transformer and does not require parameter update or architecture change. Shift-AET extracts alignments from an additional alignment module which is tightly integrated into Transformer and trained in isolation with supervision from symmetrized Shift-Att alignments. Experiments on three publicly available datasets demonstrate that both methods perform better than their corresponding neural baselines and Shift-AET significantly outperforms GIZA++ by 1.4-4.8 AER points.