mlsanityData to 7 Oct 2026

Attention Flows are Shapley Value Explanations

arXiv 2105.14652 ↗First posted 31 May 2021cs.CLcited by 30

Not on the map. Why?

Abstract (arXiv)

Shapley Values, a solution to the credit assignment problem in cooperative game theory, are a popular type of explanation in machine learning, having been used to explain the importance of features, embeddings, and even neurons. In NLP, however, leave-one-out and attention-based explanations still predominate. Can we draw a connection between these different methods? We formally prove that -- save for the degenerate case -- attention weights and leave-one-out values cannot be Shapley Values. textitAttention flow is a post-processed variant of attention weights obtained by running the max-flow algorithm on the attention graph. Perhaps surprisingly, we prove that attention flows are indeed Shapley Values, at least at the layerwise level. Given the many desirable theoretical qualities of Shapley Values -- which has driven their adoption among the ML community -- we argue that NLP practitioners should, when possible, adopt attention flow explanations alongside more traditional ones.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Builds on 16 of its 20 references that are in the map. First posted on arXiv 31 May 2021.

Builds on

16 of 20 references.

References · 16

  1. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  2. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  3. 2017SmoothGrad: removing noise by adding noisecited by 1,097
  4. 2019Attention is not Explanationcited by 916
  5. 2019Are Sixteen Heads Really Better than One?cited by 803
  6. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  7. 2020Quantifying Attention Flow in Transformerscited by 759
  8. 2019Attention is not not Explanationcited by 629
  9. 2019Data Shapley: Equitable Valuation of Data for Machine Learningcited by 617
  10. 2019Is Attention Interpretable?cited by 387
  11. 2019Revealing the Dark Secrets of BERTcited by 300
  12. 2016Understanding Neural Networks through Representation Erasurecited by 223
4 more
  1. 2018Pathologies of Neural Models Make Interpretations Difficultcited by 140
  2. 2020Neuron Shapley: Discovering the Responsible Neuronscited by 98
  3. 2013Bounding the Estimation Error of Sampling-based Shapley Value Approximationcited by 56
  4. 2016"What is Relevant in a Text Document?": An Interpretable Machine Learning Approachcited by 53

Built on by

Cited by year

Citing papers per year: 2022: 10; 2023: 7; 2024: 7; 2025: 6.

30 papers.

Sort:

By area:

  1. 2023Explainability for Large Language Models: A Surveycited by 331
  2. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 129
  3. 2023Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMscited by 121
  4. 2022Algorithms to estimate Shapley value feature attributionscited by 114
  5. 2023Towards a Mechanistic Interpretation of Multi-Step Reasoning Capabilities of Language Modelscited by 50
  6. 2022Learning to Estimate Shapley Values with Vision Transformerscited by 44
  7. 2022Rethinking Attention-Model Explainability through Faithfulness Violation Testcited by 43
  8. 2023Neuron-Level Knowledge Attribution in Large Language Modelscited by 37
  9. 2023VISIT: Visualizing and Interpreting the Semantic Information Flow of Transformerscited by 32
  10. 2024Steering Language Models with Game-Theoretic Solverscited by 28
  11. 2024Explainable Generative AI (GenXAI): A Survey, Conceptualization, and Research Agendacited by 22
  12. 2022Diagnosing AI Explanation Methods with Folk Concepts of Behaviorcited by 15
18 more
  1. 2022A Song of (Dis)agreement: Evaluating the Evaluation of Explainable Artificial Intelligence in Natural Language Processingcited by 13
  2. 2024Compositional Video Generation as Flow Equalizationcited by 12
  3. 2024Attention Meets Post-hoc Interpretability: A Mathematical Perspectivecited by 11
  4. 2022On the Importance of Clinical Notes in Multi-modal Learning for EHR Datacited by 10
  5. 2023Morphosyntactic probing of multilingual BERT modelscited by 8
  6. 2022IsoVec: Controlling the Relative Isomorphism of Word Embedding Spacescited by 7
  7. 2024RORA: Robust Free-Text Rationale Evaluationcited by 4
  8. 2025Generalized Attention Flow: Feature Attribution for Transformer Models via Maximum Flowcited by 3
  9. 2025Utility-inspired Reward Transformations Improve Reinforcement Learning Training of Language Modelscited by 2
  10. 2025Which Words Matter Most in Zero-Shot Prompts?cited by 2
  11. 2022Weakly Supervised Learning Significantly Reduces the Number of Labels Required for Intracranial Hemorrhage Detection on Head CTcited by 1
  12. 2024Using Shapley interactions to understand how models use structurecited by 1
  13. 2024Investigating Idiomaticity in Word Representationscited by 1
  14. 2022Attention Flows for General Transformerscited by 0
  15. 2023Game Theory Solutions in Sensor-Based Human Activity Recognition: A Reviewcited by 0
  16. 2025Causally-informed Deep Learning towards Explainable and Generalizable Outcomes Prediction in Critical Carecited by 0
  17. 2025OrdShap: Feature Position Importance for Sequential Black-Box Modelscited by 0
  18. 2025Integrating attention into explanation frameworks for language and vision transformerscited by 0