# Why Attentions May Not Be Interpretable?

arXiv: https://arxiv.org/abs/2006.05656 · HTML: https://mlsanity.com/paper/2006.05656/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Jun 2020.

## Built on by (most cited first, 30 of 41)

- [Explainability for Large Language Models: A Survey](https://mlsanity.com/paper/2309.01029/) (2023)
- [Interpretable Graph Neural Networks for Connectome-Based Brain Disorder Analysis](https://mlsanity.com/paper/2207.00813/) (2022)
- [Rethinking Attention-Model Explainability through Faithfulness Violation Test](https://mlsanity.com/paper/2201.12114/) (2022)
- [Neuron-Level Knowledge Attribution in Large Language Models](https://mlsanity.com/paper/2312.12141/) (2023)
- [Local Interpretations for Explainable Natural Language Processing: A Survey](https://mlsanity.com/paper/2103.11072/) (2021)
- [Interpretable Geometric Deep Learning via Learnable Randomness Injection](https://mlsanity.com/paper/2210.16966/) (2022)
- [Towards Trustworthy AI: A Review of Ethical and Robust Large Language Models](https://mlsanity.com/paper/2407.13934/) (2024)
- [Autoformalizing Natural Language to First-Order Logic: A Case Study in Logical Fallacy Detection](https://mlsanity.com/paper/2405.02318/) (2024)
- [A Comprehensive Survey on Self-Interpretable Neural Networks](https://mlsanity.com/paper/2501.15638/) (2025)
- [ClassActionPrediction: A Challenging Benchmark for Legal Judgment Prediction of Class Action Cases in the US](https://mlsanity.com/paper/2211.00582/) (2022)
- [Transformers in Reinforcement Learning: A Survey](https://mlsanity.com/paper/2307.05979/) (2023)
- [Interpreting and Exploiting Functional Specialization in Multi-Head Attention under Multi-task Learning](https://mlsanity.com/paper/2310.10318/) (2023)
- [All Circuits Lead to Rome: Rethinking Functional Anisotropy in Circuit and Sheaf Discovery for LLMs](https://mlsanity.com/paper/2605.12671/) (2026)
- [Analyzing Semantic Change through Lexical Replacements](https://mlsanity.com/paper/2404.18570/) (2024)
- [Evaluating Visual Explanations of Attention Maps for Transformer-based Medical Imaging](https://mlsanity.com/paper/2503.09535/) (2025)
- [Plausible Extractive Rationalization through Semi-Supervised Entailment Signal](https://mlsanity.com/paper/2402.08479/) (2024)
- [Short-circuiting Shortcuts: Mechanistic Investigation of Shortcuts in Text Classification](https://mlsanity.com/paper/2505.06032/) (2025)
- [Continuous Diagnosis and Prognosis by Controlling the Update Process of Deep Neural Networks](https://mlsanity.com/paper/2210.02719/) (2022)
- [Learning to Select Prototypical Parts for Interpretable Sequential Data Modeling](https://mlsanity.com/paper/2212.03396/) (2022)
- [Manifoldron: Direct Space Partition via Manifold Discovery](https://mlsanity.com/paper/2201.05279/) (2022)
- [A Novel Convolutional Neural Network Architecture with a Continuous Symmetry](https://mlsanity.com/paper/2308.01621/) (2023)
- [Insights Into the Inner Workings of Transformer Models for Protein Function Prediction](https://mlsanity.com/paper/2309.03631/) (2023)
- [Does Faithfulness Conflict with Plausibility? An Empirical Study in Explainable AI across NLP Tasks](https://mlsanity.com/paper/2404.00140/) (2024)
- [Infinite-Dimensional Feature Interaction](https://mlsanity.com/paper/2405.13972/) (2024)
- [Can Hessian-Based Insights Support Fault Diagnosis in Attention-based Models?](https://mlsanity.com/paper/2506.07871/) (2025)
- [Evaluating Explanation Methods for Vision-and-Language Navigation](https://mlsanity.com/paper/2310.06654/) (2023)
- [Discovering robust biomarkers of psychiatric disorders from resting-state functional MRI via graph neural networks: A systematic review](https://mlsanity.com/paper/2405.00577/) (2024)
- [Towards Understanding Sensitive and Decisive Patterns in Explainable AI: A Case Study of Model Interpretation in Geometric Deep Learning](https://mlsanity.com/paper/2407.00849/) (2024)
- [Regularization, Semi-supervision, and Supervision for a Plausible Attention-Based Explanation](https://mlsanity.com/paper/2501.12775/) (2025)
- [TestAug: A Framework for Augmenting Capability-based NLP Tests](https://mlsanity.com/paper/2210.08097/) (2022)

## Abstract (arXiv)

Attention-based methods have played important roles in model interpretations, where the calculated attention weights are expected to highlight the critical parts of inputs~(e.g., keywords in sentences). However, recent research found that attention-as-importance interpretations often do not work as we expected. For example, learned attention weights sometimes highlight less meaningful tokens like "[SEP]", ",", and ".", and are frequently uncorrelated with other feature importance indicators like gradient-based measures. A recent debate over whether attention is an explanation or not has drawn considerable interest. In this paper, we demonstrate that one root cause of this phenomenon is the combinatorial shortcuts, which means that, in addition to the highlighted parts, the attention weights themselves may carry extra information that could be utilized by downstream models after attention layers. As a result, the attention weights are no longer pure importance indicators. We theoretically analyze combinatorial shortcuts, design one intuitive experiment to show their existence, and propose two methods to mitigate this issue. We conduct empirical studies on attention-based interpretation models. The results show that the proposed methods can effectively improve the interpretability of attention mechanisms.