mlsanityData to 7 Oct 2026

XAI for Transformers: Better Explanations through Conservative Propagation

arXiv 2202.07304 ↗First posted 15 Feb 2022cs.LGcited by 100

Not on the map. Why?

Abstract (arXiv)

Transformers have become an important workhorse of machine learning, with numerous applications. This necessitates the development of reliable methods for increasing their transparency. Multiple interpretability methods, often based on gradient information, have been proposed. We show that the gradient in a Transformer reflects the function only locally, and thus fails to reliably identify the contribution of input features to the prediction. We identify Attention Heads and LayerNorm as main reasons for such unreliable explanations and propose a more stable way for propagation through these layers. Our proposal, which can be seen as a proper extension of the well-established LRP method to Transformers, is shown both theoretically and empirically to overcome the deficiency of a simple gradient-based approach, and achieves state-of-the-art explanation performance on a broad range of Transformer models and datasets.

Key facts. Cited by 100 papers in the map (data through 7 Oct 2026). Builds on 44 of its 60 references that are in the map. First posted on arXiv 15 Feb 2022.

Builds on

44 of 60 references.

References · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  4. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  7. 2017Learning Important Features Through Propagating Activation Differencescited by 1,454
  8. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  9. 2017MoleculeNet: A Benchmark for Molecular Machine Learningcited by 1,037
  10. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  11. 2018A Survey Of Methods For Explaining Black Box Modelscited by 970
  12. 2019Attention is not Explanationcited by 916
32 more
  1. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  2. 2020Quantifying Attention Flow in Transformerscited by 759
  3. 2019GNNExplainer: Generating Explanations for Graph Neural Networkscited by 735
  4. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  5. 2020Transformer Interpretability Beyond Attention Visualizationcited by 620
  6. 2019Graph Transformer Networkscited by 520
  7. 2016Geometric deep learning on graphs and manifolds using mixture model CNNscited by 468
  8. 2020Parameterized Explainer for Graph Neural Networkcited by 423
  9. 2019Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Settingcited by 346
  10. 2017Methods for Interpreting and Understanding Deep Neural Networkscited by 342
  11. 2021Do Transformers Really Perform Bad for Graph Representation?cited by 318
  12. 2021Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformerscited by 316
  13. 2019Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Themcited by 268
  14. 2018Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systemscited by 234
  15. 2020GraphLIME: Local Interpretable Model Explanations for Graph Neural Networkscited by 209
  16. 2020A Survey of the State of Explainable AI for Natural Language Processingcited by 206
  17. 2019Full-Gradient Representation for Neural Network Visualizationcited by 189
  18. 2020A Diagnostic Study of Explainability Techniques for Text Classificationcited by 158
  19. 2018Pathologies of Neural Models Make Interpretations Difficultcited by 140
  20. 2020Higher-Order Explanations of Graph Neural Networks via Relevant Walkscited by 138
  21. 2020Molecule Attention Transformercited by 92
  22. 2019AllenNLP Interpret: A Framework for Explaining Predictions of NLP Modelscited by 74
  23. 2019Perturbation Sensitivity Analysis to Detect Unintended Model Biasescited by 74
  24. 2021Gophormer: Ego-Graph Transformer for Node Classificationcited by 63
  25. 2019TWEETQA: A Social Media Focused Question Answering Datasetcited by 55
  26. 2014Extraction of Salient Sentences from Labelled Documentscited by 50
  27. 2020Hierarchical Pre-training for Sequence Labelling in Spoken Dialogcited by 36
  28. 2021Fast Axiomatic Attribution for Neural Networkscited by 35
  29. 2019Explaining and Interpreting LSTMscited by 23
  30. 2021Relative Importance in Sentence Processingcited by 22
  31. 2021On Explaining Your Explanations of BERT: An Empirical Study with Sequence Classificationcited by 21
  32. 2020Graph-Aware Transformer: Is Attention All Graphs Need?cited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 14; 2024: 31; 2025: 29; 2026: 20.

100 papers.

Sort:

By area:

  1. 2023Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directionscited by 123
  2. 2024AttnLRP: Attention-Aware Layer-Wise Relevance Propagation for Transformerscited by 116
  3. 2024The Hidden Attention of Mamba Modelscited by 88
  4. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 83
  5. 2023Adversarial attacks and defenses in explainable artificial intelligence: A surveycited by 47
  6. 2024Explainable Artificial Intelligence: A Survey of Needs, Techniques, Applications, and Future Directioncited by 46
  7. 2022So3krates: Equivariant attention for interactions on arbitrary length-scales in molecular systemscited by 39
  8. 2024MambaLRP: Explaining Selective State Space Sequence Modelscited by 31
  9. 2025Token Activation Map to Visually Explain Multimodal LLMscited by 30
  10. 2023On the Independence of Association Bias and Empirical Fairness in Language Modelscited by 29
  11. 2022Disentangled Explanations of Neural Network Predictions by Finding Relevant Subspacescited by 25
  12. 2024The Clever Hans Effect in Unsupervised Learningcited by 24
38 more
  1. 2024Explainable Generative AI (GenXAI): A Survey, Conceptualization, and Research Agendacited by 22
  2. 2022Do Transformer Models Show Similar Attention Patterns to Task-Specific Human Gaze?cited by 20
  3. 2024Token Transformation Matters: Towards Faithful Post-hoc Explanation for Vision Transformercited by 19
  4. 2024On the Faithfulness of Vision Transformer Explanationscited by 19
  5. 2026Language Model Circuits Are Sparse in the Neuron Basiscited by 18
  6. 2024xMIL: Insightful Explanations for Multiple Instance Learning in Histopathologycited by 15
  7. 2023Being Right for Whose Right Reasons?cited by 12
  8. 2025Neural ODE Transformers: Analyzing Internal Dynamics and Adaptive Fine-tuningcited by 12
  9. 2023Rather a Nurse than a Physician -- Contrastive Explanations under Investigationcited by 11
  10. 2024Explaining Modern Gated-Linear RNNs via a Unified Implicit Attention Formulationcited by 11
  11. 2024Pruning By Explaining Revisited: Optimizing Attribution Methods to Prune CNNs and Transformerscited by 11
  12. 2025The Atlas of In-Context Learning: How Attention Heads Shape In-Context Retrieval Augmentationcited by 11
  13. 2023Class-Discriminative Attention Maps for Vision Transformerscited by 10
  14. 2025A Close Look at Decomposition-based XAI-Methods for Transformer Language Modelscited by 10
  15. 2025RelP: Faithful and Efficient Circuit Discovery in Language Models via Relevance Patchingcited by 10
  16. 2022Shapley Head Pruning: Identifying and Removing Interference in Multilingual Transformerscited by 9
  17. 2023Preemptively Pruning Clever-Hans Strategies in Deep Neural Networkscited by 9
  18. 2024Explaining Text Similarity in Transformer Modelscited by 9
  19. 2025GMAR: Gradient-Driven Multi-Head Attention Rollout for Vision Transformer Interpretabilitycited by 9
  20. 2025Capturing Polysemanticity with PRISM: A Multi-Concept Feature Description Frameworkcited by 9
  21. 2024Better Explain Transformers by Illuminating Important Informationcited by 8
  22. 2023On the Interplay between Fairness and Explainabilitycited by 7
  23. 2024Layer-Wise Relevance Propagation with Conservation Property for ResNetcited by 7
  24. 2024Towards Symbolic XAI -- Explanation Through Human Understandable Logical Relationships Between Featurescited by 7
  25. 2025Evaluating the Effectiveness of XAI Techniques for Encoder-Based Language Modelscited by 7
  26. 2025A Unified Framework with Novel Metrics for Evaluating the Effectiveness of XAI Techniques in LLMscited by 7
  27. 2025Towards Desiderata-Driven Design of Visual Counterfactual Explainerscited by 7
  28. 2024Unveiling and Manipulating Prompt Influence in Large Language Modelscited by 6
  29. 2024Progressive Inference: Explaining Decoder-Only Sequence Classification Models Using Intermediate Predictionscited by 6
  30. 2024Accurate Explanation Model for Image Classifiers using Class Association Embeddingcited by 6
  31. 2025Revisiting LRP: Positional Attribution as the Missing Ingredient for Transformer Explainabilitycited by 6
  32. 2024ASAP: Interpretable Analysis and Summarization of AI-generated Image Patterns at Scalecited by 5
  33. 2024LibraGrad: Balancing Gradient Flow for Universally Better Vision Transformer Attributionscited by 5
  34. 2025Evidence Recomposition and Predictive Context Residualization for Visual Attribution in Multimodal Large Language Modelscited by 5
  35. 2023A Survey of Explainable AI in Deep Visual Modeling: Methods and Metricscited by 4
  36. 2024Enhancing Efficiency in Vision Transformer Networks: Design Techniques and Insightscited by 4
  37. 2025Correcting Gradient-Based Circuit Localization via Interaction-Aware Backpropagationcited by 4
  38. 2025Quantifying Cross-Attention Interaction in Transformers for Interpreting TCR-pMHC Bindingcited by 4

50 most cited of 100 All 100 (JSON)