mlsanityData to 7 Oct 2026

AttnLRP: Attention-Aware Layer-Wise Relevance Propagation for Transformers

arXiv 2402.05602 ↗First posted 8 Feb 2024cs.CL cs.AI cs.CVcited by 116

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Large Language Models are prone to biased predictions and hallucinations, underlining the paramount importance of understanding their model-internal reasoning process. However, achieving faithful attributions for the entirety of a black-box transformer model and maintaining computational efficiency is an unsolved challenge. By extending the Layer-wise Relevance Propagation attribution method to handle attention layers, we address these challenges effectively. While partial solutions exist, our method is the first to faithfully and holistically attribute not only input but also latent representations of transformer models with the computational efficiency similar to a single backward pass. Through extensive evaluations against existing methods on LLaMa 2, Mixtral 8x7b, Flan-T5 and vision transformer architectures, we demonstrate that our proposed approach surpasses alternative methods in terms of faithfulness and enables the understanding of latent representations, opening up the door for concept-based explanations. We provide an LRP library at https://github.com/rachtibat/LRP-eXplains-Transformers.

Key facts. Cited by 116 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 59 of its 74 references that are in the map. First posted on arXiv 8 Feb 2024.

Builds on

59 of 74 references.

References · 59

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  6. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  7. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  8. 2016Layer Normalizationcited by 5,579
  9. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  10. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  11. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  12. 2017Axiomatic Attribution for Deep Networkscited by 3,350
38 more
  1. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  2. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  3. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  4. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  5. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  6. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  7. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  8. 2017Learning Important Features Through Propagating Activation Differencescited by 1,454
  9. 2019Root Mean Square Layer Normalizationcited by 1,425
  10. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  11. 2017SmoothGrad: removing noise by adding noisecited by 1,097
  12. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  13. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  14. 2020Quantifying Attention Flow in Transformerscited by 759
  15. 2019Attention is not not Explanationcited by 629
  16. 2020Transformer Interpretability Beyond Attention Visualizationcited by 620
  17. 2017Interpretable Explanations of Black Boxes by Meaningful Perturbationcited by 611
  18. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  19. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  20. 2020Captum: A unified and generic model interpretability library for PyTorchcited by 542
  21. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  22. 2015Evaluating the visualization of what a Deep Neural Network has learnedcited by 416
  23. 2021Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformerscited by 316
  24. 2015Explaining NonLinear Classification Decisions with Deep Taylor Decompositioncited by 248
  25. 2016Layer-wise Relevance Propagation for Neural Networks with Local Renormalization Layerscited by 197
  26. 2016Synthesizing the preferred inputs for neurons in neural networks via deep generator networkscited by 181
  27. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 175
  28. 2022Quantus: An Explainable AI Toolkit for Responsible Evaluation of Neural Network Explanations and Beyondcited by 171
  29. 2018Local Rule-Based Explanations of Black Box Decision Systemscited by 165
  30. 2017The Shattered Gradients Problem: If resnets are the answer, then what is the question?cited by 156
  31. 2017Learning to Generate Reviews and Discovering Sentimentcited by 147
  32. 2022A Review of Sparse Expert Models in Deep Learningcited by 147
  33. 2018Explaining Image Classifiers by Counterfactual Generationcited by 144
  34. 2022From Attribution Maps to Human-Understandable Explanations through Concept Relevance Propagationcited by 141
  35. 2020Ground Truth Evaluation of Neural Network Explanations with CLEVR-XAIcited by 125
  36. 2022XAI for Transformers: Better Explanations through Conservative Propagationcited by 100
  37. 2023Neurons in Large Language Models: Dead, N-gram, Positionalcited by 88
  38. 2020Analyzing the Source and Target Contributions to Predictions in Neural Machine Translationcited by 71

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 43; 2026: 61.

116 papers.

Sort:

By area:

  1. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 83
  2. 2025The Hidden Dimensions of LLM Alignment: A Multi-Dimensional Analysis of Orthogonal Safety Directionscited by 38
  3. 2025Steering CLIP's vision transformer with sparse autoencoderscited by 33
  4. 2024MambaLRP: Explaining Selective State Space Sequence Modelscited by 31
  5. 2025Token Activation Map to Visually Explain Multimodal LLMscited by 30
  6. 2026Language Model Circuits Are Sparse in the Neuron Basiscited by 18
  7. 2025Interpretable Text Embeddings and Text Similarity Explanation: A Surveycited by 12
  8. 2024Pruning By Explaining Revisited: Optimizing Attribution Methods to Prune CNNs and Transformerscited by 11
  9. 2025The Atlas of In-Context Learning: How Attention Heads Shape In-Context Retrieval Augmentationcited by 11
  10. 2025Attention (as Discrete-Time Markov) Chainscited by 11
  11. 2025A Close Look at Decomposition-based XAI-Methods for Transformer Language Modelscited by 10
  12. 2025Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safetycited by 10
38 more
  1. 2025RelP: Faithful and Efficient Circuit Discovery in Language Models via Relevance Patchingcited by 10
  2. 2025Attention Consistency for LLMs Explanationcited by 10
  3. 2025CoE: Chain-of-Explanation via Automatic Visual Concept Circuit Description and Polysemanticity Quantificationcited by 9
  4. 2024Towards Symbolic XAI -- Explanation Through Human Understandable Logical Relationships Between Featurescited by 7
  5. 2025Unraveling LLM Jailbreaks Through Safety Knowledge Neuronscited by 7
  6. 2024Sparse, Efficient and Explainable Data Attribution with DualXDAcited by 6
  7. 2024Unlocking Large Language Model's Planning Capabilities with Maximum Diversity Fine-tuningcited by 6
  8. 2025Revisiting LRP: Positional Attribution as the Missing Ingredient for Transformer Explainabilitycited by 6
  9. 2024Aggregated Attributions for Explanatory Analysis of 3D Segmentation Modelscited by 5
  10. 2024LibraGrad: Balancing Gradient Flow for Universally Better Vision Transformer Attributionscited by 5
  11. 2025GLIMPSE: Holistic Cross-Modal Explainability for Large Vision-Language Modelscited by 5
  12. 2025Position: We Need An Algorithmic Understanding of Generative AIcited by 5
  13. 2025Evidence Recomposition and Predictive Context Residualization for Visual Attribution in Multimodal Large Language Modelscited by 5
  14. 2025Understanding LLMs' Cross-Lingual Context Retrieval: How Good It Is And Where It Comes Fromcited by 4
  15. 2025Correcting Gradient-Based Circuit Localization via Interaction-Aware Backpropagationcited by 4
  16. 2025Quantifying Cross-Attention Interaction in Transformers for Interpreting TCR-pMHC Bindingcited by 4
  17. 2025Causal Interpretation of Sparse Autoencoder Features in Visioncited by 4
  18. 2025Circuit Insights: Towards Interpretability Beyond Activationscited by 4
  19. 2025GorillaWatch: An Automated System for In-the-Wild Gorilla Re-Identification and Population Monitoringcited by 4
  20. 2024A Hierarchical Language Model For Interpretable Graph Reasoningcited by 3
  21. 2025VITAL: More Understandable Feature Visualization through Distribution Alignment and Relevant Information Flowcited by 3
  22. 2025Pairwise Matching of Intermediate Representations for Fine-grained Explainabilitycited by 3
  23. 2025RePPL: Recalibrating Perplexity by Uncertainty in Semantic Propagation and Language Generation for Explainable QA Hallucination Detectioncited by 3
  24. 2025Response Uncertainty and Probe Modeling: Two Sides of the Same Coin in LLM Interpretability?cited by 3
  25. 2025Attribution-Guided Pruning for Insight and Control: Circuit Discovery and Targeted Correction in Small-scale LLMscited by 3
  26. 2026DAVE: Distribution-aware Attribution via ViT Gradient Decompositioncited by 3
  27. 2026Beyond Accuracy: Robustness, Interpretability and Expressiveness of EEG Foundation Modelscited by 3
  28. 2026Looking Is Not Picking: An Attention-Segment Account of Tool-Selection Failures in LLM Agentscited by 3
  29. 2024DLBacktrace: A Model Agnostic Explainability for any Deep Learning Modelscited by 2
  30. 2025A Data-Driven Diffusion-based Approach for Audio Deepfake Explanationscited by 2
  31. 2025Explainable time-series forecasting with sampling-free SHAP for Transformerscited by 2
  32. 2026MIRROR: A Multi-Agent Framework with Iterative Adaptive Revision and Hierarchical Retrieval for Optimization Modeling in Operations Researchcited by 2
  33. 2026From Heuristic Selection to Automated Algorithm Design: LLMs Benefit from Strong Priorscited by 2
  34. 2026Normalized Relevance Measure as a Unifying Framework to Explain Neural Network Latent Structurescited by 2
  35. 2024A Systematic Comparison between Extractive Self-Explanations and Human Rationales in Text Classificationcited by 1
  36. 2025Large Language Models as Attribution Regularizers for Efficient Model Trainingcited by 1
  37. 2025Extracting Explainable Dates From Medical Images By Reverse-Engineering UNIX Timestampscited by 1
  38. 2025A Review of Developmental Interpretability in Large Language Modelscited by 1

50 most cited of 116 All 116 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Is Random Attention Sufficient for Sequence Modeling? Disentangling Trainable Components in the Transformer
  2. 2023Attention Alignment and Flexible Positional Embeddings Improve Transformer Length Extrapolation
  3. 2023Jump to Conclusions: Short-Cutting Transformers With Linear Transformations
  4. 2025SCOUT: Toward Sub-Quadratic Attention via Segment Compression for Optimized Utility in Transformers
  5. 2026Revisiting the Shape Convention of Transformer Language Models
  6. 2025The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
  7. 2024RecurFormer: Not All Transformer Heads Need Self-Attention
  8. 2024Separating Tongue from Thought: Activation Patching Reveals Language-Agnostic Concept Representations in Transformers