Data to 7 Oct 2026

Attention is not Explanation

arXiv 1902.10186 ↗First posted 26 Feb 2019cs.CL cs.AIcited by 916

Not on the map. Why?

Cited by 34 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Attention mechanisms have seen wide adoption in neural NLP models. In addition to improving predictive performance, these are often touted as affording transparency: models equipped with attention provide a distribution over attended-to input units, and this is often presented (at least implicitly) as communicating the relative importance of inputs. However, it is unclear what relationship exists between attention weights and model outputs. In this work, we perform extensive experiments across a variety of NLP tasks that aim to assess the degree to which attention weights provide meaningful `explanations' for predictions. We find that they largely do not. For example, learned attention weights are frequently uncorrelated with gradient-based measures of feature importance, and one can identify very different attention distributions that nonetheless yield equivalent predictions. Our findings show that standard attention modules do not provide meaningful explanations and should not be treated as though they do. Code for all experiments is available at https://github.com/successar/AttentionExplanation.

Key facts. Cited by 916 papers in the map (data through 7 Oct 2026). First posted on arXiv 26 Feb 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 149; 2022: 135; 2023: 113; 2024: 121; 2025: 132; 2026: 266.

916 papers.

Sort:

By area:

2021Improving language models by retrieving from trillions of tokenscited by 1,278
2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
2021A Survey on Automated Fact-Checkingcited by 376
2023Explainability for Large Language Models: A Surveycited by 331
2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Drivingcited by 231
2021Trustworthy AI: From Principles to Practicescited by 206
2021Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyondcited by 177
2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 176
2022Interpretable and Generalizable Graph Learning via Stochastic Attention Mechanismcited by 171
2022A Review on Language Models as Knowledge Basescited by 156
2021Post-hoc Interpretability for Neural NLP: A Surveycited by 155

38 more

2022Comparative layer-wise analysis of self-supervised speech modelscited by 155
2023LingoQA: Visual Question Answering for Autonomous Drivingcited by 144
2021Benchmarking and Survey of Explanation Methods for Black Box Modelscited by 136
2022Towards Faithful Model Explanation in NLP: A Surveycited by 129
2022Text and Patterns: For Effective Chain of Thought, It Takes Two to Tangocited by 128
2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 126
2023Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMscited by 121
2023AI Transparency in the Age of LLMs: A Human-Centered Research Roadmapcited by 119
2021Interpretable Deep Learning: Interpretation, Interpretability, Trustworthiness, and Beyondcited by 116
2023Mechanistically analyzing the effects of fine-tuning on procedurally defined taskscited by 110
2022XAI for Transformers: Better Explanations through Conservative Propagationcited by 100
2021Do Feature Attribution Methods Correctly Attribute Features?cited by 98
2024ContextCite: Attributing Model Generation to Contextcited by 93
2023Language Model Behavior: A Comprehensive Surveycited by 92
2024Evolutionary Computation in the Era of Large Language Model: Survey and Roadmapcited by 89
2024The Hidden Attention of Mamba Modelscited by 88
2021The Out-of-Distribution Problem in Explainability and Search Methods for Feature Importance Explanationscited by 86
2021Towards a Science of Human-AI Decision Making: A Survey of Empirical Studiescited by 85
2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 83
2023Tell Your Model Where to Attend: Post-hoc Attention Steering for LLMscited by 82
2021Explainability of deep vision-based autonomous driving systems: Review and challengescited by 80
2021Explanations in Autonomous Driving: A Surveycited by 80
2021ILDC for CJPE: Indian Legal Documents Corpus for Court Judgment Prediction and Explanationcited by 78
2023Inseq: An Interpretability Toolkit for Sequence Generation Modelscited by 78
2021Paragraph-level Rationale Extraction through Regularization: A case study on European Court of Human Rights Casescited by 77
2021A Survey on Stance Detection for Mis- and Disinformation Identificationcited by 75
2021Is Attention Better Than Matrix Decomposition?cited by 75
2023Exposing Attention Glitches with Flip-Flop Language Modelingcited by 74
2021A Practical Survey on Faster and Lighter Transformerscited by 73
2023Eight Things to Know about Large Language Modelscited by 73
2021Explaining Time Series Predictions with Dynamic Maskscited by 72
2021Vision-and-Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformerscited by 71
2023Large Language Models and Knowledge Graphs: Opportunities and Challengescited by 71
2021A Survey on Interpretable Reinforcement Learningcited by 70
2021"Will You Find These Shortcuts?" A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text Classificationcited by 66
2022Generating Literal and Implied Subquestions to Fact-check Complex Claimscited by 66
2022A General Survey on Attention Mechanisms in Deep Learningcited by 64
2025RTBAS: Defending LLM Agents Against Prompt Injection and Privacy Leakagecited by 64

50 most cited of 916 All 916 (JSON)