# Interrogating the Explanatory Power of Attention in Neural Machine Translation

arXiv: https://arxiv.org/abs/1910.00139 · HTML: https://mlsanity.com/paper/1910.00139/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Sep 2019.

## Built on by (most cited first, 7 of 7)

- [Wait-info Policy: Balancing Source and Target at Information Level for Simultaneous Machine Translation](https://mlsanity.com/paper/2210.11220/) (2022)
- [Effective Attention Sheds Light On Interpretability](https://mlsanity.com/paper/2105.08855/) (2021)
- [Should We Attend More or Less? Modulating Attention for Fairness](https://mlsanity.com/paper/2305.13088/) (2023)
- [CipherDAug: Ciphertext based Data Augmentation for Neural Machine Translation](https://mlsanity.com/paper/2204.00665/) (2022)
- [Explaining with Contrastive Phrasal Highlighting: A Case Study in Assisting Humans to Detect Translation Differences](https://mlsanity.com/paper/2312.01582/) (2023)
- [Automated and Explainable Ontology Extension Based on Deep Learning: A Case Study in the Chemical Domain](https://mlsanity.com/paper/2109.09202/) (2021)
- [An empirical analysis of phrase-based and neural machine translation](https://mlsanity.com/paper/2103.03108/) (2021)

## Abstract (arXiv)

Attention models have become a crucial component in neural machine translation (NMT). They are often implicitly or explicitly used to justify the model's decision in generating a specific token but it has not yet been rigorously established to what extent attention is a reliable source of information in NMT. To evaluate the explanatory power of attention for NMT, we examine the possibility of yielding the same prediction but with counterfactual attention models that modify crucial aspects of the trained attention model. Using these counterfactual attention mechanisms we assess the extent to which they still preserve the generation of function and content words in the translation process. Compared to a state of the art attention model, our counterfactual attention models produce 68% of function words and 21% of content words in our German-English dataset. Our experiments demonstrate that attention models by themselves cannot reliably explain the decisions made by a NMT model.