# Staying True to Your Word: (How) Can Attention Become Explanation?

arXiv: https://arxiv.org/abs/2005.09379 · HTML: https://mlsanity.com/paper/2005.09379/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 May 2020.

## Built on by (most cited first, 18 of 18)

- [Towards Faithful Model Explanation in NLP: A Survey](https://mlsanity.com/paper/2209.11326/) (2022)
- [Order in the Court: Explainable AI Methods Prone to Disagreement](https://mlsanity.com/paper/2105.03287/) (2021)
- [Understanding Interlocking Dynamics of Cooperative Rationalization](https://mlsanity.com/paper/2110.13880/) (2021)
- [Is Sparse Attention more Interpretable?](https://mlsanity.com/paper/2106.01087/) (2021)
- [Improving the Faithfulness of Attention-based Explanations with Task-specific Information for Text Classification](https://mlsanity.com/paper/2105.02657/) (2021)
- [Enjoy the Salience: Towards Better Transformer-based Faithful Explanations with Word Salience](https://mlsanity.com/paper/2108.13759/) (2021)
- [Incorporating Attribution Importance for Improving Faithfulness Metrics](https://mlsanity.com/paper/2305.10496/) (2023)
- [A Song of (Dis)agreement: Evaluating the Evaluation of Explainable Artificial Intelligence in Natural Language Processing](https://mlsanity.com/paper/2205.04559/) (2022)
- [Effective Attention Sheds Light On Interpretability](https://mlsanity.com/paper/2105.08855/) (2021)
- [Explaining with Contrastive Phrasal Highlighting: A Case Study in Assisting Humans to Detect Translation Differences](https://mlsanity.com/paper/2312.01582/) (2023)
- [SalKG: Learning From Knowledge Graph Explanations for Commonsense Reasoning](https://mlsanity.com/paper/2104.08793/) (2021)
- [Higher-order Neural Additive Models: An Interpretable Machine Learning Model with Feature Interactions](https://mlsanity.com/paper/2209.15409/) (2022)
- [Follow-up Attention: An Empirical Study of Developer and Neural Model Code Exploration](https://mlsanity.com/paper/2210.05506/) (2022)
- [Bi-Orthogonal Factor Decomposition for Vision Transformers](https://mlsanity.com/paper/2601.05328/) (2026)
- [A survey on bias in machine learning research](https://mlsanity.com/paper/2308.11254/) (2023)
- [On Explaining with Attention Matrices](https://mlsanity.com/paper/2410.18541/) (2024)
- [A Study of the Plausibility of Attention between RNN Encoders in Natural Language Inference](https://mlsanity.com/paper/2501.13735/) (2025)
- [Easy to Decide, Hard to Agree: Reducing Disagreements Between Saliency Methods](https://mlsanity.com/paper/2211.08369/) (2022)

## Abstract (arXiv)

The attention mechanism has quickly become ubiquitous in NLP. In addition to improving performance of models, attention has been widely used as a glimpse into the inner workings of NLP models. The latter aspect has in the recent years become a common topic of discussion, most notably in work of Jain and Wallace, 2019; Wiegreffe and Pinter, 2019. With the shortcomings of using attention weights as a tool of transparency revealed, the attention mechanism has been stuck in a limbo without concrete proof when and whether it can be used as an explanation. In this paper, we provide an explanation as to why attention has seen rightful critique when used with recurrent networks in sequence classification tasks. We propose a remedy to these issues in the form of a word level objective and our findings give credibility for attention to provide faithful interpretations of recurrent models.