mlsanityData to 7 Oct 2026

Grad-CAM for Vision Transformers: A Systematic Taxonomy and Audit of Methodological Ambiguity in Explainable AI

arXiv 2608.05258 ↗First posted 5 Aug 2026cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Gradient-weighted Class Activation Mapping (Grad-CAM) is widely used to visualize model decisions, but it was originally formulated for convolutional neural networks, where spatial feature maps and channel dimensions have clear architectural meanings. Vision Transformers (ViTs) do not provide the same structure, instead representing images through tokens, attention, residual streams, and multimodal interactions. This paper presents a systematic taxonomy and literature audit of how Grad-CAM and related methods are adapted, justified, and reported for ViT-based architectures. From an initial search of more than 550 papers, we identify 175 papers that apply Grad-CAM or Grad-CAM-adjacent methods to ViTs. We find that most papers do not provide a full mathematical or implementation-level account of how Grad-CAM is adapted to transformer representations. To characterize this gap, we introduce a descriptive taxonomy of ViT Grad-CAM adaptations that makes explicit the feature locations, gradient targets, spatial reconstruction steps, and aggregation choices that are often left implicit. This taxonomy is not intended to prescribe a single correct adaptation, but to clarify the range of methodological choices being made. The study shows that Grad-CAM on ViTs is often treated as a trivial extension of CNN-based Grad-CAM, despite requiring nontrivial choices that affect rigor, reproducibility, and interpretation.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 144 of its 228 references that are in the map. First posted on arXiv 5 Aug 2026.

Builds on

144 of 228 references.

References · 144

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  7. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  9. 2015Learning Deep Features for Discriminative Localizationcited by 2,430
  10. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  11. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  12. 2017Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networkscited by 936
38 more
  1. 2020Quantifying Attention Flow in Transformerscited by 759
  2. 2020Transformer Interpretability Beyond Attention Visualizationcited by 620
  3. 2019Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networkscited by 575
  4. 2021MetaFormer Is Actually What You Need for Visioncited by 523
  5. 2021TransReID: Transformer-based Object Re-Identificationcited by 432
  6. 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 281
  7. 2021ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Biascited by 216
  8. 2021General Facial Representation Learning in a Visual-Linguistic Mannercited by 193
  9. 2019Full-Gradient Representation for Neural Network Visualizationcited by 189
  10. 2023Forgery-aware Adaptive Transformer for Generalizable Synthetic Image Detectioncited by 180
  11. 2020Axiom-based Grad-CAM: Towards Accurate Visualization and Explanation of CNNscited by 151
  12. 2023Spike-driven Transformercited by 141
  13. 2021IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformerscited by 127
  14. 2022CLIP is Also an Efficient Segmenter: A Text-Driven Approach for Weakly Supervised Semantic Segmentationcited by 118
  15. 2022From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Modelscited by 114
  16. 2024EMR-Merging: Tuning-Free High-Performance Model Mergingcited by 108
  17. 2023TALL: Thumbnail Layout for Deepfake Video Detectioncited by 100
  18. 2022Inception Transformercited by 99
  19. 2022Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Trainingcited by 88
  20. 2021Delving Deep into the Generalization of Vision Transformers under Distribution Shiftscited by 85
  21. 2023InceptionNeXt: When Inception Meets ConvNeXtcited by 84
  22. 2023BlackVIP: Black-Box Visual Prompting for Robust Transfer Learningcited by 70
  23. 2023CLIP as RNN: Segment Countless Visual Concepts without Training Endeavorcited by 69
  24. 2022Delving into Masked Autoencoders for Multi-Label Thorax Disease Classificationcited by 65
  25. 2022Multi-modal Alignment using Representation Codebookcited by 63
  26. 2023Zero-shot Referring Image Segmentation with Global-Local Context Featurescited by 61
  27. 2023Med-UniC: Unifying Cross-Lingual Medical Vision-Language Pre-Training by Diminishing Biascited by 57
  28. 2022A Comprehensive Study of Image Classification Model Sensitivity to Foregrounds, Backgrounds, and Visual Attributescited by 56
  29. 2022MARLIN: Masked Autoencoder for facial video Representation LearnINgcited by 55
  30. 2023FunnyBirds: A Synthetic Vision Dataset for a Part-Based Analysis of Explainable AI Methodscited by 49
  31. 2024Matryoshka Query Transformer for Large Vision-Language Modelscited by 49
  32. 2024BigGait: Learning Gait Representation You Want by Large Vision Modelscited by 44
  33. 2024IMDL-BenCo: A Comprehensive Benchmark and Codebase for Image Manipulation Detection & Localizationcited by 42
  34. 2022Delving into Sequential Patches for Deepfake Detectioncited by 41
  35. 2023ArGue: Attribute-Guided Prompt Tuning for Vision-Language Modelscited by 39
  36. 2024SimVG: A Simple Framework for Visual Grounding with Decoupled Multi-modal Fusioncited by 36
  37. 2023Vision Transformers with Mixed-Resolution Tokenizationcited by 35
  38. 2023Visual Explanations of Image-Text Representations via Multi-Modal Information Bottleneck Attributioncited by 34

50 most cited of 144 All 144 (JSON)

Built on by

Not cited yet.