Data to 13 Oct 2026

Transformer Interpretability Beyond Attention Visualization

arXiv 2012.09838 ↗First posted 17 Dec 2020cs.CVcited by 623

Not on the map. Why?

Cited by 53 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Self-attention techniques, and specifically Transformers, are dominating the field of text processing and are becoming increasingly popular in computer vision classification tasks. In order to visualize the parts of the image that led to a certain classification, existing methods either rely on the obtained attention maps or employ heuristic propagation along the attention graph. In this work, we propose a novel way to compute relevancy for Transformer networks. The method assigns local relevance based on the Deep Taylor Decomposition principle and then propagates these relevancy scores through the layers. This propagation involves attention layers and skip connections, which challenge existing methods. Our solution is based on a specific formulation that is shown to maintain the total relevancy across layers. We benchmark our method on very recent visual Transformer networks, as well as on a text classification problem, and demonstrate a clear advantage over the existing explainability methods.

Key facts. Cited by 623 papers in the map (data through 13 Oct 2026). First posted on arXiv 17 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 32; 2022: 88; 2023: 120; 2024: 108; 2025: 127; 2026: 148.

623 papers.

Sort:

By area:

2021Transformers in Vision: A Surveycited by 1,010
2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 852
2023Explainability for Large Language Models: A Surveycited by 333
2023DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Datacited by 327
2021Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformerscited by 318
2022Transformers in Medical Imaging: A Surveycited by 245
2023Break-A-Scene: Extracting Multiple Concepts from a Single Imagecited by 230
2022SpaText: Spatio-Textual Representation for Controllable Image Generationcited by 217
2022CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigationcited by 194
2023Interpreting CLIP's Image Representation via Text-Based Decompositioncited by 194
2022CLIP-ReID: Exploiting Vision-Language Model for Image Re-Identification without Concrete Text Labelscited by 178
2021Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformercited by 175

38 more

2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMscited by 175
2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruningcited by 166
2022SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentationcited by 164
2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 163
2022Can Machines Help Us Answering Question 16 in Datasheets, and In Turn Reflecting on Inappropriate Content?cited by 154
2022FlexiViT: One Model for All Patch Sizescited by 152
2021All Tokens Matter: Token Labeling for Training Better Vision Transformerscited by 144
2022Towards Faithful Model Explanation in NLP: A Surveycited by 129
2021IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformerscited by 127
2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 127
2021Interpretable Deep Learning: Interpretation, Interpretability, Trustworthiness, and Beyondcited by 118
2024AttnLRP: Attention-Aware Layer-Wise Relevance Propagation for Transformerscited by 117
2021Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factorscited by 107
2022Retrieval Augmented Classification for Long-Tail Visual Recognitioncited by 106
2022XAI for Transformers: Better Explanations through Conservative Propagationcited by 100
2023A Closer Look at the Explainability of Contrastive Language-Image Pre-trainingcited by 96
2022"This is my unicorn, Fluffy": Personalizing frozen vision-language representationscited by 95
2022Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Modelscited by 95
2024The Hidden Attention of Mamba Modelscited by 89
2021TransFER: Learning Relation-aware Facial Expression Representations with Transformerscited by 85
2022A graph-transformer for whole slide image classificationcited by 84
2024ZigMa: A DiT-style Zigzag Mamba Diffusion Modelcited by 84
2021TransMix: Attend to Mix for Vision Transformerscited by 79
2023Time Series as Images: Vision Transformer for Irregularly Sampled Time Seriescited by 79
2022What do Vision Transformers Learn? A Visual Explorationcited by 76
2024DeCo: Decoupling Token Compression from Semantic Abstraction in Multimodal Large Language Modelscited by 74
2024DreamLIP: Language-Image Pre-training with Long Captionscited by 72
2023X-Pruner: eXplainable Pruning for Vision Transformerscited by 67
2023Turning large language models into cognitive modelscited by 67
2022Vision Transformer Slimming: Multi-Dimension Searching in Continuous Optimization Spacecited by 66
2021KVT: k-NN Attention for Boosting Vision Transformerscited by 65
2022Delving into Masked Autoencoders for Multi-Label Thorax Disease Classificationcited by 65
2022Video Transformers: A Surveycited by 63
2022Measuring the Mixing of Contextual Information in the Transformercited by 63
2022ProtoPFormer: Concentrating on Prototypical Parts in Vision Transformers for Interpretable Image Recognitioncited by 63
2023Word-As-Image for Semantic Typographycited by 63
2023A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Futurecited by 59
2022VL-InterpreT: An Interactive Visualization Tool for Interpreting Vision-Language Transformerscited by 53

50 most cited of 623 All 623 (JSON)