Data to 14 Oct 2026

Attention is not not Explanation

arXiv 1908.04626 ↗First posted 13 Aug 2019cs.CLcited by 636

Not on the map. Why?

Cited by 23 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Attention mechanisms play a central role in NLP systems, especially within recurrent neural network (RNN) models. Recently, there has been increasing interest in whether or not the intermediate representations offered by these modules may be used to explain the reasoning for a model's prediction, and consequently reach insights regarding the model's decision-making process. A recent paper claims that `Attention is not Explanation' (Jain and Wallace, 2019). We challenge many of the assumptions underlying this work, arguing that such a claim depends on one's definition of explanation, and that testing it needs to take into account all elements of the model, using a rigorous experimental design. We propose four alternative tests to determine when/whether attention can be used as explanation: a simple uniform-weights baseline; a variance calibration based on multiple random seed runs; a diagnostic framework using frozen weights from pretrained models; and an end-to-end adversarial attention training protocol. Each allows for meaningful interpretation of attention mechanisms in RNN models. We show that even when reliable adversarial distributions can be found, they don't perform well on the simple diagnostic, indicating that prior work does not disprove the usefulness of attention mechanisms for explainability.

Key facts. Cited by 636 papers in the map (data through 14 Oct 2026). First posted on arXiv 13 Aug 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 121; 2022: 104; 2023: 89; 2024: 83; 2025: 92; 2026: 147.

636 papers.

Sort:

By area:

2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 878
2021A Survey on Automated Fact-Checkingcited by 378
2023Explainability for Large Language Models: A Surveycited by 333
2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
2021Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Modelscited by 222
2021Trustworthy AI: From Principles to Practicescited by 207
2023Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretabilitycited by 205
2022Comparative layer-wise analysis of self-supervised speech modelscited by 158
2022A Review on Language Models as Knowledge Basescited by 157
2021Post-hoc Interpretability for Neural NLP: A Surveycited by 156
2024MambaOut: Do We Really Need Mamba for Vision?cited by 143
2022Towards Faithful Model Explanation in NLP: A Surveycited by 130

38 more

2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 129
2021Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processingcited by 126
2023AI Transparency in the Age of LLMs: A Human-Centered Research Roadmapcited by 123
2024AttnLRP: Attention-Aware Layer-Wise Relevance Propagation for Transformerscited by 120
2021Interpretable Deep Learning: Interpretation, Interpretability, Trustworthiness, and Beyondcited by 118
2024Rethinking Interpretability in the Era of Large Language Modelscited by 111
2021Do Feature Attribution Methods Correctly Attribute Features?cited by 98
2023Tell Your Model Where to Attend: Post-hoc Attention Steering for LLMscited by 91
2021NODE-GAM: Neural Generalized Additive Model for Interpretable Deep Learningcited by 88
2021The Out-of-Distribution Problem in Explainability and Search Methods for Feature Importance Explanationscited by 87
2021Towards a Science of Human-AI Decision Making: A Survey of Empirical Studiescited by 86
2021Explanations in Autonomous Driving: A Surveycited by 81
2021Paragraph-level Rationale Extraction through Regularization: A case study on European Court of Human Rights Casescited by 77
2021Is Attention Better Than Matrix Decomposition?cited by 77
2021Contrastive Explanations for Model Interpretabilitycited by 75
2021A Practical Survey on Faster and Lighter Transformerscited by 73
2021A Survey on Interpretable Reinforcement Learningcited by 73
2024Attention Score is not All You Need for Token Importance Indicator in KV Cache Reduction: Value Also Matterscited by 73
2025RTBAS: Defending LLM Agents Against Prompt Injection and Privacy Leakagecited by 69
2022Measuring the Mixing of Contextual Information in the Transformercited by 65
2022A General Survey on Attention Mechanisms in Deep Learningcited by 64
2022Video Transformers: A Surveycited by 63
2022Looking for a Needle in a Haystack: A Comprehensive Study of Hallucinations in Neural Machine Translationcited by 61
2021OperA: Attention-Regularized Transformers for Surgical Phase Recognitioncited by 60
2021Unmasking the Mask -- Evaluating Social Biases in Masked Language Modelscited by 60
2022PINTO: Faithful Language Reasoning Using Prompt-Generated Rationalescited by 56
2022Entropy-based Attention Regularization Frees Unintended Bias Mitigation from Listscited by 55
2021MATE: Multi-view Attention for Table Transformer Efficiencycited by 54
2024Explainable AI for Safe and Trustworthy Autonomous Driving: A Systematic Reviewcited by 53
2024Conformal Prediction for Natural Language Processing: A Surveycited by 53
2023Self-Verification Improves Few-Shot Clinical Information Extractioncited by 52
2021Attention, please! A survey of Neural Attention Models in Deep Learningcited by 51
2021Representation learning for neural population activity with Neural Data Transformerscited by 50
2022Investigating Explainability of Generative AI for Code through Scenario-based Designcited by 47
2021Biomedical Question Answering: A Survey of Approaches and Challengescited by 46
2021A Review on Explainability in Multimodal Deep Neural Netscited by 46
2022Learning to Estimate Shapley Values with Vision Transformerscited by 45
2024Interpreting and Improving Large Language Models in Arithmetic Calculationcited by 45

50 most cited of 636 All 636 (JSON)