# Attention cannot be an Explanation

arXiv: https://arxiv.org/abs/2201.11194 · HTML: https://mlsanity.com/paper/2201.11194/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 51 references that are in the map.
- First posted on arXiv 26 Jan 2022.

## Builds on (most cited first, 26 of 26)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [Attention is not Explanation](https://mlsanity.com/paper/1902.10186/) (2019)
- [Counterfactual Visual Explanations](https://mlsanity.com/paper/1904.07451/) (2019)
- [Explanations based on the Missing: Towards Contrastive Explanations with Pertinent Negatives](https://mlsanity.com/paper/1802.07623/) (2018)
- [Metrics for Explainable AI: Challenges and Prospects](https://mlsanity.com/paper/1812.04608/) (2018)
- [Interpretable Convolutional Neural Networks](https://mlsanity.com/paper/1710.00935/) (2017)
- [Visualizing and Understanding Recurrent Networks](https://mlsanity.com/paper/1506.02078/) (2015)
- [The Bayesian Case Model: A Generative Approach for Case-Based Reasoning and Prototype Classification](https://mlsanity.com/paper/1503.01161/) (2015)
- [Words aren't enough, their order matters: On the Robustness of Grounding Visual Referring Expressions](https://mlsanity.com/paper/2005.01655/) (2020)
- [Natural Language Interaction with Explainable AI Models](https://mlsanity.com/paper/1903.05720/) (2019)
- [X-ToM: Explaining with Theory-of-Mind for Gaining Justified Human Trust](https://mlsanity.com/paper/1909.06907/) (2019)
- [Influence in Classification via Cooperative Game Theory](https://mlsanity.com/paper/1505.00036/) (2015)
- [LSTMVis: A Tool for Visual Analysis of Hidden State Dynamics in Recurrent Neural Networks](https://mlsanity.com/paper/1606.07461/) (2016)
- [Effective Representation to Capture Collaboration Behaviors between Explainer and User](https://mlsanity.com/paper/2201.03147/) (2022)
- [Question Generation for Evaluating Cross-Dataset Shifts in Multi-modal Grounding](https://mlsanity.com/paper/2201.09639/) (2022)
- [Discourse Analysis for Evaluating Coherence in Video Paragraph Captions](https://mlsanity.com/paper/2201.06207/) (2022)
- [Discourse Parsing in Videos: A Multi-modal Appraoch](https://mlsanity.com/paper/1903.02252/) (2019)

## Built on by (most cited first, 4 of 4)

- [Bi-Orthogonal Factor Decomposition for Vision Transformers](https://mlsanity.com/paper/2601.05328/) (2026)
- [Multidimensional Perceptron for Efficient and Explainable Long Text Classification](https://mlsanity.com/paper/2304.01638/) (2023)
- [Concept Graph Convolutions: Message Passing in the Concept Space](https://mlsanity.com/paper/2604.20082/) (2026)
- [A Unifying View of Attention Sinks: From Mechanisms to Architectural Interventions](https://mlsanity.com/paper/2606.08105/) (2026)

## Abstract (arXiv)

Attention based explanations (viz. saliency maps), by providing interpretability to black box models such as deep neural networks, are assumed to improve human trust and reliance in the underlying models. Recently, it has been shown that attention weights are frequently uncorrelated with gradient-based measures of feature importance. Motivated by this, we ask a follow-up question: "Assuming that we only consider the tasks where attention weights correlate well with feature importance, how effective are these attention based explanations in increasing human trust and reliance in the underlying models?". In other words, can we use attention as an explanation? We perform extensive human study experiments that aim to qualitatively and quantitatively assess the degree to which attention based explanations are suitable in increasing human trust and reliance. Our experiment results show that attention cannot be used as an explanation.