mlsanityData to 7 Oct 2026

Rethinking Attention-Model Explainability through Faithfulness Violation Test

arXiv 2201.12114 ↗First posted 28 Jan 2022cs.LG cs.CL cs.CVcited by 43

Not on the map. Why?

Abstract (arXiv)

Attention mechanisms are dominating the explainability of deep models. They produce probability distributions over the input, which are widely deemed as feature-importance indicators. However, in this paper, we find one critical limitation in attention explanations: weakness in identifying the polarity of feature impact. This would be somehow misleading -- features with higher attention weights may not faithfully contribute to model predictions; instead, they can impose suppression effects. With this finding, we reflect on the explainability of current attention-based techniques, such as AttentioodotGradient and LRP-based attention explanations. We first propose an actionable diagnostic methodology (henceforth faithfulness violation test) to measure the consistency between explanation weights and the impact polarity. Through the extensive experiments, we then show that most tested explanation methods are unexpectedly hindered by the faithfulness violation issue, especially the raw attention. Empirical analyses on the factors affecting violation issues further provide useful observations for adopting explanation methods in attention models.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Builds on 51 of its 74 references that are in the map. First posted on arXiv 28 Jan 2022.

Builds on

51 of 74 references.

References · 51

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  4. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  5. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  6. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  9. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  10. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  11. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  12. 2015A large annotated corpus for learning natural language inferencecited by 1,837
38 more
  1. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  2. 2017Learning Important Features Through Propagating Activation Differencescited by 1,454
  3. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  4. 2018Sanity Checks for Saliency Mapscited by 1,149
  5. 2017SmoothGrad: removing noise by adding noisecited by 1,097
  6. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  7. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  8. 2019Attention is not Explanationcited by 916
  9. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  10. 2019Are Sixteen Heads Really Better than One?cited by 803
  11. 2020Quantifying Attention Flow in Transformerscited by 759
  12. 2019Attention is not not Explanationcited by 629
  13. 2020Transformer Interpretability Beyond Attention Visualizationcited by 620
  14. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 590
  15. 2019ERASER: A Benchmark to Evaluate Rationalized NLP Modelscited by 523
  16. 2018A Benchmark for Interpretability Methods in Deep Neural Networkscited by 467
  17. 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 423
  18. 2019Is Attention Interpretable?cited by 387
  19. 2014Recurrent Models of Visual Attentioncited by 344
  20. 2021Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformerscited by 316
  21. 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 309
  22. 2015Stacked Attention Networks for Image Question Answeringcited by 273
  23. 2019Understanding Deep Networks via Extremal Perturbations and Smooth Maskscited by 257
  24. 2016Understanding Neural Networks through Representation Erasurecited by 223
  25. 2016Layer-wise Relevance Propagation for Neural Networks with Local Renormalization Layerscited by 197
  26. 2017Visualizing Deep Neural Network Decisions: Prediction Difference Analysiscited by 166
  27. 2020A Diagnostic Study of Explainability Techniques for Text Classificationcited by 158
  28. 2020Self-Attention Attribution: Interpreting Information Interactions Inside Transformercited by 124
  29. 2019Sanity Checks for Saliency Metricscited by 119
  30. 2019On Identifiability in Transformerscited by 115
  31. 2018L-Shapley and C-Shapley: Efficient Model Interpretation for Structured Datacited by 100
  32. 2019When Explanations Lie: Why Many Modified BP Attributions Failcited by 92
  33. 2021The Out-of-Distribution Problem in Explainability and Search Methods for Feature Importance Explanationscited by 86
  34. 2020Towards Transparent and Explainable Attention Modelscited by 47
  35. 2020Why Attentions May Not Be Interpretable?cited by 41
  36. 2017Show, Ask, Attend, and Answer: A Strong Baseline For Visual Question Answeringcited by 31
  37. 2021Attention Flows are Shapley Value Explanationscited by 30
  38. 2021Improving the Faithfulness of Attention-based Explanations with Task-specific Information for Text Classificationcited by 29

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 8; 2024: 16; 2025: 8; 2026: 5.

43 papers.

Sort:

By area:

  1. 2023Explainability for Large Language Models: A Surveycited by 331
  2. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 129
  3. 2022Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Modelscited by 95
  4. 2023Towards a Mechanistic Interpretation of Multi-Step Reasoning Capabilities of Language Modelscited by 50
  5. 2023Faithful Explanations of Black-box NLP Models Using LLM-generated Counterfactualscited by 44
  6. 2022Optimizing Relevance Maps of Vision Transformers Improves Robustnesscited by 41
  7. 2024On the Challenges and Opportunities in Generative AIcited by 31
  8. 2022EATFormer: Improving Vision Transformer Inspired by Evolutionary Algorithmcited by 27
  9. 2024On the Faithfulness of Vision Transformer Explanationscited by 19
  10. 2024F-Fidelity: A Robust Framework for Faithfulness Evaluation of Explainable AIcited by 17
  11. 2023Interpretable Multimodal Misinformation Detection with Logic Reasoningcited by 16
  12. 2023A Review of ChatGPT Applications in Education, Marketing, Software Engineering, and Healthcare: Benefits, Drawbacks, and Research Directionscited by 12
31 more
  1. 2024TELLER: A Trustworthy Framework for Explainable, Generalizable and Controllable Fake News Detectioncited by 10
  2. 2025ExDDV: A New Dataset for Explainable Deepfake Detection in Videocited by 10
  3. 2023Interpreting and Exploiting Functional Specialization in Multi-Head Attention under Multi-task Learningcited by 9
  4. 2024Explainable Natural Language Processing for Corporate Sustainability Analysiscited by 9
  5. 2022An Attention Matrix for Every Decision: Faithfulness-based Arbitration Among Multiple Attention-Based Interpretations of Transformers in Text Classificationcited by 6
  6. 2024Towards Quantitative Evaluation of Explainable AI Methods for Deepfake Detectioncited by 6
  7. 2024Beyond Accuracy: Ensuring Correct Predictions With Correct Rationalescited by 6
  8. 2024RORA: Robust Free-Text Rationale Evaluationcited by 4
  9. 2025Heart2Mind: Human-Centered Contestable Psychiatric Disorder Diagnosis System using Wearable ECG Monitorscited by 4
  10. 2025A Meaningful Perturbation Metric for Evaluating Explainability Methodscited by 3
  11. 2023Generic Attention-model Explainability by Weighted Relevance Accumulationcited by 2
  12. 2024Faithfulness and the Notion of Adversarial Sensitivity in NLP Explanationscited by 2
  13. 2024AdaptGrad: Adaptive Sampling to Reduce Noisecited by 2
  14. 2024Beyond Accuracy: On the Effects of Fine-tuning Towards Vision-Language Model's Prediction Rationalitycited by 2
  15. 2025Argument-Based Consistency in Toxicity Explanations of LLMscited by 2
  16. 2026Concept-Guided Fine-Tuning: Steering ViTs away from Spurious Correlations to Improve Robustnesscited by 2
  17. 2022Truthful Meta-Explanations for Local Interpretability of Machine Learning Modelscited by 1
  18. 2024Advancing Attribution-Based Neural Network Explainability through Relative Absolute Magnitude Layer-Wise Relevance Propagation and Multi-Component Evaluationcited by 1
  19. 2025Quantifying Modality Contributions via Disentangling Multimodal Representationscited by 1
  20. 2023SpecXAI -- Spectral interpretability of Deep Learning Modelscited by 0
  21. 2024eXplainable Bayesian Multi-Perspective Generative Retrievalcited by 0
  22. 2024Demystifying the Physics of Deep Reinforcement Learning-Based Autonomous Vehicle Decision-Makingcited by 0
  23. 2024Axiomatization of Gradient Smoothing in Neural Networkscited by 0
  24. 2024Exploiting Latent Linearity in LLMs Improves Explainable Molecular Representation Learningcited by 0
  25. 2025Hierarchical Attention Network for Interpretable ECG-based Heart Disease Classificationcited by 0
  26. 2025An Experimental Study on Generating Plausible Textual Explanations for Video Summarizationcited by 0
  27. 2025PruneGCRN: Minimizing and explaining spatio-temporal problems through node pruningcited by 0
  28. 2026KTCF: Actionable Recourse in Knowledge Tracing via Counterfactual Explanations for Educationcited by 0
  29. 2026Interpretability in Deep Time Series Models Demands Semantic Alignmentcited by 0
  30. 2026Gradient-Skipping Relevance Propagation for Efficient Explainability of Vision Transformerscited by 0
  31. 2026Beyond Attention: Signed Integrated Gradients Attribution in a BiomeGPT-Style Microbiome Transformercited by 0