mlsanityData to 7 Oct 2026

Attention Meets Post-hoc Interpretability: A Mathematical Perspective

arXiv 2402.03485 ↗First posted 5 Feb 2024stat.ML cs.CL cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

Attention-based architectures, in particular transformers, are at the heart of a technological revolution. Interestingly, in addition to helping obtain state-of-the-art results on a wide range of applications, the attention mechanism intrinsically provides meaningful insights on the internal behavior of the model. Can these insights be used as explanations? Debate rages on. In this paper, we mathematically study a simple attention-based architecture and pinpoint the differences between post-hoc and attention-based explanations. We show that they provide quite different results, and that, despite their limitations, post-hoc methods are capable of capturing more useful insights than merely examining the attention weights.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 41 of its 58 references that are in the map. First posted on arXiv 5 Feb 2024.

Builds on

41 of 58 references.

References · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  4. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  7. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  8. 2019Attention is not Explanationcited by 916
  9. 2022Transformers learn in-context by gradient descentcited by 714
  10. 2019Attention is not not Explanationcited by 629
  11. 2020Transformer Interpretability Beyond Attention Visualizationcited by 620
  12. 2015Bidirectional LSTM-CRF Models for Sequence Taggingcited by 544
29 more
  1. 2019Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methodscited by 440
  2. 2019Is Attention Interpretable?cited by 387
  3. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  4. 2019Analyzing the Structure of Attention in a Transformer Language Modelcited by 241
  5. 2015Visualizing and Understanding Neural Models in NLPcited by 199
  6. 2020A Diagnostic Study of Explainability Techniques for Text Classificationcited by 158
  7. 2021Inductive Biases and Variable Creation in Self-Attention Mechanismscited by 144
  8. 2021Benchmarking and Survey of Explanation Methods for Black Box Modelscited by 136
  9. 2020The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?cited by 122
  10. 2019On Identifiability in Transformerscited by 115
  11. 2014Long Short-Term Memory Based Recurrent Neural Network Architectures for Large Vocabulary Speech Recognitioncited by 111
  12. 2019Attention in Natural Language Processingcited by 100
  13. 2022Vision Transformers provably learn spatial structurecited by 92
  14. 2021The Out-of-Distribution Problem in Explainability and Search Methods for Feature Importance Explanationscited by 86
  15. 2023How Do Transformers Learn Topic Structure: Towards a Mechanistic Understandingcited by 85
  16. 2023Transformers as Support Vector Machinescited by 67
  17. 2014Extraction of Salient Sentences from Labelled Documentscited by 50
  18. 2023What can a Single Attention Layer Learn? A Study Through the Random Features Lenscited by 39
  19. 2021Order in the Court: Explainable AI Methods Prone to Disagreementcited by 36
  20. 2021Attention Flows are Shapley Value Explanationscited by 30
  21. 2023Transformers are uninterpretable with myopic methods: a case study with bounded Dyck grammarscited by 30
  22. 2019Evaluating Recurrent Neural Network Explanationscited by 29
  23. 2019Generating Token-Level Explanations for Natural Language Inferencecited by 28
  24. 2020An Analysis of LIME for Text Datacited by 25
  25. 2022A Song of (Dis)agreement: Evaluating the Evaluation of Explainable Artificial Intelligence in Natural Language Processingcited by 13
  26. 2021Effective Attention Sheds Light On Interpretabilitycited by 11
  27. 2022A Sea of Words: An In-Depth Analysis of Anchors for Text Datacited by 6
  28. 2022An Attention Matrix for Every Decision: Faithfulness-based Arbitration Among Multiple Attention-Based Interpretations of Transformers in Text Classificationcited by 6
  29. 2023Faithful and Robust Local Interpretability for Textual Predictionscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 5; 2026: 4.

11 papers.

  1. 2025Training a Utility-based Retriever Through Shared Context Attribution for Retrieval-Augmented Language Modelscited by 13
  2. 2025Masks Can Be Distracting: On Context Comprehension in Diffusion Language Modelscited by 7
  3. 2024Discovering robust biomarkers of psychiatric disorders from resting-state functional MRI via graph neural networks: A systematic reviewcited by 1
  4. 2025Feature Attribution from First Principlescited by 1
  5. 2025There is More to Attention: Statistical Filtering Enhances Explanations in Vision Transformerscited by 1
  6. 2024Impact of Adversarial Attacks on Deep Learning Model Explainabilitycited by 0
  7. 2025The Role of Feature Interactions in Graph-based Tabular Deep Learningcited by 0
  8. 2026Memory Retrieval in Transformers: Insights from The Encoding Specificity Principlecited by 0
  9. 2026LEMON: Local Explanations via Modality-aware OptimizatioNcited by 0
  10. 2026Contribution Weights: A Geometrical Analysis of Self-Attention Transformerscited by 0
  11. 2026How Do LLMs Read Bug Reports? An Empirical Study of Attention in LLMs for Automated Program Repaircited by 0