# Better Explain Transformers by Illuminating Important Information

arXiv: https://arxiv.org/abs/2401.09972 · HTML: https://mlsanity.com/paper/2401.09972/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 42 references that are in the map.
- First posted on arXiv 18 Jan 2024.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [Attention is not Explanation](https://mlsanity.com/paper/1902.10186/) (2019)
- [Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned](https://mlsanity.com/paper/1905.09418/) (2019)
- [Quantifying Attention Flow in Transformers](https://mlsanity.com/paper/2005.00928/) (2020)
- [Transformer Interpretability Beyond Attention Visualization](https://mlsanity.com/paper/2012.09838/) (2020)
- [Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers](https://mlsanity.com/paper/2103.15679/) (2021)
- [Revealing the Dark Secrets of BERT](https://mlsanity.com/paper/1908.08593/) (2019)
- [Explaining NonLinear Classification Decisions with Deep Taylor Decomposition](https://mlsanity.com/paper/1512.02479/) (2015)
- [Full-Gradient Representation for Neural Network Visualization](https://mlsanity.com/paper/1905.00780/) (2019)
- [A Diagnostic Study of Explainability Techniques for Text Classification](https://mlsanity.com/paper/2009.13295/) (2020)
- [Pathologies of Neural Models Make Interpretations Difficult](https://mlsanity.com/paper/1804.07781/) (2018)
- [XAI for Transformers: Better Explanations through Conservative Propagation](https://mlsanity.com/paper/2202.07304/) (2022)
- [AllenNLP Interpret: A Framework for Explaining Predictions of NLP Models](https://mlsanity.com/paper/1909.09251/) (2019)
- [Perturbation Sensitivity Analysis to Detect Unintended Model Biases](https://mlsanity.com/paper/1910.04210/) (2019)
- [Measuring the Mixing of Contextual Information in the Transformer](https://mlsanity.com/paper/2203.04212/) (2022)
- [Incorporating Residual and Normalization Layers into Analysis of Masked Language Models](https://mlsanity.com/paper/2109.07152/) (2021)
- [GlobEnc: Quantifying Global Token Attribution by Incorporating the Whole Encoder Layer in Transformers](https://mlsanity.com/paper/2205.03286/) (2022)
- [Grad-SAM: Explaining Transformers via Gradient Self-Attention Maps](https://mlsanity.com/paper/2204.11073/) (2022)

## Built on by (most cited first, 8 of 8)

- [Explaining Length Bias in LLM-Based Preference Evaluations](https://mlsanity.com/paper/2407.01085/) (2024)
- [Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safety](https://mlsanity.com/paper/2506.05451/) (2025)
- [M3OOD: Automatic Selection of Multimodal OOD Detectors](https://mlsanity.com/paper/2508.11936/) (2025)
- [From Attribution to Abstention: Training-Free Attention-Based Auditing for Clinical Summarization](https://mlsanity.com/paper/2601.16397/) (2026)
- [XAI-enhanced Comparative Opinion Mining via Aspect-based Scoring and Semantic Reasoning](https://mlsanity.com/paper/2603.01212/) (2026)
- [OmniTrace: A Unified Framework for Generation-Time Attribution in Omni-Modal LLMs](https://mlsanity.com/paper/2604.13073/) (2026)
- [Learning to Explain: Supervised Token Attribution from Transformer Attention Patterns](https://mlsanity.com/paper/2601.14112/) (2026)
- [Scaling Attention Head Analysis via Gradient-Based Attribution in Context-Aware Machine Translation](https://mlsanity.com/paper/2609.28117/) (2026)

## Abstract (arXiv)

Transformer-based models excel in various natural language processing (NLP) tasks, attracting countless efforts to explain their inner workings. Prior methods explain Transformers by focusing on the raw gradient and attention as token attribution scores, where non-relevant information is often considered during explanation computation, resulting in confusing results. In this work, we propose highlighting the important information and eliminating irrelevant information by a refined information flow on top of the layer-wise relevance propagation (LRP) method. Specifically, we consider identifying syntactic and positional heads as important attention heads and focus on the relevance obtained from these important heads. Experimental results demonstrate that irrelevant information does distort output attribution scores and then should be masked during explanation computation. Compared to eight baselines on both classification and question-answering datasets, our method consistently outperforms with over 3% to 33% improvement on explanation metrics, providing superior explanation performance. Our anonymous code repository is available at: https://github.com/LinxinS97/Mask-LRP