mlsanityData to 14 Oct 2026

Is Attention Interpretable?

arXiv 1906.03731 ↗First posted 9 Jun 2019cs.CLcited by 389

Not on the map. Why?

Abstract (arXiv)

Attention mechanisms have recently boosted performance on a range of NLP tasks. Because attention layers explicitly weight input components' representations, it is also often assumed that attention can be used to identify information that models found important (e.g., specific contextualized word tokens). We test whether that assumption holds by manipulating attention weights in already-trained text classification models and analyzing the resulting differences in their predictions. While we observe some ways in which higher attention weights correlate with greater impact on model predictions, we also find many ways in which this does not hold, i.e., where gradient-based rankings of attention weights better predict their effects than their magnitudes. We conclude that while attention noisily predicts input components' overall importance to a model, it is by no means a fail-safe indicator.

Key facts. Cited by 389 papers in the map (data through 14 Oct 2026). First posted on arXiv 9 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 78; 2022: 86; 2023: 71; 2024: 59; 2025: 40; 2026: 55.

389 papers.

Sort:

By area:

  1. 2021A Survey on Automated Fact-Checkingcited by 378
  2. 2023Explainability for Large Language Models: A Surveycited by 333
  3. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  4. 2022From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AIcited by 302
  5. 2022What the DAAM: Interpreting Stable Diffusion Using Cross Attentioncited by 236
  6. 2022Interpretable and Generalizable Graph Learning via Stochastic Attention Mechanismcited by 171
  7. 2022A Review on Language Models as Knowledge Basescited by 157
  8. 2021Post-hoc Interpretability for Neural NLP: A Surveycited by 156
  9. 2021TransFG: A Transformer Architecture for Fine-grained Recognitioncited by 132
  10. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 130
  11. 2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 129
  12. 2023Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMscited by 124
38 more
  1. 2021Interpretable Deep Learning: Interpretation, Interpretability, Trustworthiness, and Beyondcited by 118
  2. 2023Mechanistically analyzing the effects of fine-tuning on procedurally defined taskscited by 115
  3. 2024ContextCite: Attributing Model Generation to Contextcited by 98
  4. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  5. 2023Attention Sorting Combats Recency Bias In Long Context Language Modelscited by 90
  6. 2023Can Large Language Models Explain Themselves? A Study of LLM-Generated Self-Explanationscited by 90
  7. 2021Towards a Science of Human-AI Decision Making: A Survey of Empirical Studiescited by 86
  8. 2021Is Attention Better Than Matrix Decomposition?cited by 77
  9. 2021A Practical Survey on Faster and Lighter Transformerscited by 73
  10. 2022Measuring the Mixing of Contextual Information in the Transformercited by 65
  11. 2022FormNet: Structural Encoding beyond Sequential Modeling in Form Document Information Extractioncited by 64
  12. 2022Video Transformers: A Surveycited by 63
  13. 2022ProtoPFormer: Concentrating on Prototypical Parts in Vision Transformers for Interpretable Image Recognitioncited by 63
  14. 2022Looking for a Needle in a Haystack: A Comprehensive Study of Hallucinations in Neural Machine Translationcited by 61
  15. 2021Counterfactuals and Causability in Explainable Artificial Intelligence: Theory, Algorithms, and Applicationscited by 59
  16. 2022Entropy-based Attention Regularization Frees Unintended Bias Mitigation from Listscited by 55
  17. 2021Attention, please! A survey of Neural Attention Models in Deep Learningcited by 51
  18. 2022Towards Opening the Black Box of Neural Machine Translation: Source and Target Interpretations of the Transformercited by 45
  19. 2022Learning to Estimate Shapley Values with Vision Transformerscited by 45
  20. 2021SelfExplain: A Self-Explaining Architecture for Neural Text Classifierscited by 44
  21. 2022GlobEnc: Quantifying Global Token Attribution by Incorporating the Whole Encoder Layer in Transformerscited by 44
  22. 2022Rethinking Attention-Model Explainability through Faithfulness Violation Testcited by 43
  23. 2023Explaining How Transformers Use Context to Build Predictionscited by 43
  24. 2024Attention in Large Language Models Yields Efficient Zero-Shot Re-Rankerscited by 43
  25. 2022Grad-SAM: Explaining Transformers via Gradient Self-Attention Mapscited by 42
  26. 2023AtmoRep: A stochastic model of atmosphere dynamics using large scale representation learningcited by 40
  27. 2022A Comparative Study of Faithfulness Metrics for Model Interpretability Methodscited by 39
  28. 2021Approximating How Single Head Attention Learnscited by 38
  29. 2021DISSECT: Disentangled Simultaneous Explanations via Concept Traversalscited by 37
  30. 2023Neuron-Level Knowledge Attribution in Large Language Modelscited by 37
  31. 2021Understanding Interlocking Dynamics of Cooperative Rationalizationcited by 36
  32. 2022Can Rationalization Improve Robustness?cited by 36
  33. 2022Going Beyond XAI: A Systematic Survey for Explanation-Guided Learningcited by 36
  34. 2023HalOmi: A Manually Annotated Benchmark for Multilingual Hallucination and Omission Detection in Machine Translationcited by 35
  35. 2021Explainable AI: current status and future directionscited by 33
  36. 2021Leveraging redundancy in attention with Reuse Transformerscited by 33
  37. 2021Deep Learning Methods for Vessel Trajectory Prediction based on Recurrent Neural Networkscited by 32
  38. 2021Local Interpretations for Explainable Natural Language Processing: A Surveycited by 31

50 most cited of 389 All 389 (JSON)