# Attention Guided CAM: Visual Explanations of Vision Transformer Guided by Self-Attention

arXiv: https://arxiv.org/abs/2402.04563 · HTML: https://mlsanity.com/paper/2402.04563/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 35 references that are in the map.
- First posted on arXiv 7 Feb 2024.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networks](https://mlsanity.com/paper/1710.11063/) (2017)
- [Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned](https://mlsanity.com/paper/1905.09418/) (2019)
- [CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification](https://mlsanity.com/paper/2103.14899/) (2021)
- [Transformer Interpretability Beyond Attention Visualization](https://mlsanity.com/paper/2012.09838/) (2020)
- [Intriguing Properties of Vision Transformers](https://mlsanity.com/paper/2105.10497/) (2021)
- [Evaluating the visualization of what a Deep Neural Network has learned](https://mlsanity.com/paper/1509.06321/) (2015)
- [Explaining NonLinear Classification Decisions with Deep Taylor Decomposition](https://mlsanity.com/paper/1512.02479/) (2015)
- [LocalViT: Analyzing Locality in Vision Transformers](https://mlsanity.com/paper/2104.05707/) (2021)
- [Layer-wise Relevance Propagation for Neural Networks with Local Renormalization Layers](https://mlsanity.com/paper/1604.00825/) (2016)
- [Are Convolutional Neural Networks or Transformers more like human vision?](https://mlsanity.com/paper/2105.07197/) (2021)
- [Informative Class Activation Maps](https://mlsanity.com/paper/2106.10472/) (2021)

## Built on by (most cited first, 17 of 17)

- [Probing the Visualization Literacy of Vision Language Models: the Good, the Bad, and the Ugly](https://mlsanity.com/paper/2504.05445/) (2025)
- [Innovative Speech-Based Deep Learning Approaches for Parkinson's Disease Classification: A Systematic Review](https://mlsanity.com/paper/2407.17844/) (2024)
- [On the Evaluation Consistency of Attribution-based Explanations](https://mlsanity.com/paper/2407.19471/) (2024)
- [Large-Scale Data-Free Knowledge Distillation for ImageNet via Multi-Resolution Data Generation](https://mlsanity.com/paper/2411.17046/) (2024)
- [QUEST: A robust attention formulation using query-modulated spherical attention](https://mlsanity.com/paper/2604.00199/) (2026)
- [Prion-ViT: Prions-Inspired Vision Transformers for Temperature prediction with Specklegrams](https://mlsanity.com/paper/2411.05836/) (2024)
- [Metric-Guided Synthesis of Class Activation Mapping](https://mlsanity.com/paper/2504.09998/) (2025)
- [Teaching Prompts to Coordinate: Hierarchical Layer-Grouped Prompt Tuning for Continual Learning](https://mlsanity.com/paper/2511.12090/) (2025)
- [LAYA: Layer-wise Attention Aggregation for Interpretable Depth-Aware Neural Networks](https://mlsanity.com/paper/2511.12723/) (2025)
- [Application of Graph Based Vision Transformers Architectures for Accurate Temperature Prediction in Fiber Specklegram Sensors](https://mlsanity.com/paper/2511.14792/) (2025)
- [What Helps---and What Hurts: Bidirectional Explanations for Vision Transformers](https://mlsanity.com/paper/2603.01605/) (2026)
- [SCAN: Visual Explanations with Self-Confidence and Analysis Networks](https://mlsanity.com/paper/2603.06523/) (2026)
- [Decision-Aware Attention Propagation for Vision Transformer Explainability](https://mlsanity.com/paper/2604.18094/) (2026)
- [Machine Collective Intelligence for Explainable Scientific Discovery](https://mlsanity.com/paper/2604.27297/) (2026)
- [Revealing the Gap in Human and VLM Scene Perception through Counterfactual Semantic Saliency](https://mlsanity.com/paper/2605.13047/) (2026)
- [Attention-Guided Saliency Maps for Interpreting Visualization Literacy in VLMs](https://mlsanity.com/paper/2607.16105/) (2026)
- [GARLIC: Graph Attention-based Relational Learning of Multivariate Time Series in Intensive Care](https://mlsanity.com/paper/2608.10969/) (2026)

## Abstract (arXiv)

Vision Transformer(ViT) is one of the most widely used models in the computer vision field with its great performance on various tasks. In order to fully utilize the ViT-based architecture in various applications, proper visualization methods with a decent localization performance are necessary, but these methods employed in CNN-based models are still not available in ViT due to its unique structure. In this work, we propose an attention-guided visualization method applied to ViT that provides a high-level semantic explanation for its decision. Our method selectively aggregates the gradients directly propagated from the classification output to each self-attention, collecting the contribution of image features extracted from each location of the input image. These gradients are additionally guided by the normalized self-attention scores, which are the pairwise patch correlation scores. They are used to supplement the gradients on the patch-level context information efficiently detected by the self-attention mechanism. This approach of our method provides elaborate high-level semantic explanations with great localization performance only with the class labels. As a result, our method outperforms the previous leading explainability methods of ViT in the weakly-supervised localization task and presents great capability in capturing the full instances of the target class object. Meanwhile, our method provides a visualization that faithfully explains the model, which is demonstrated in the perturbation comparison test.