# LeGrad: An Explainability Method for Vision Transformers via Feature Formation Sensitivity

arXiv: https://arxiv.org/abs/2404.03214 · HTML: https://mlsanity.com/paper/2404.03214/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 52 of its 59 references that are in the map.
- First posted on arXiv 4 Apr 2024.

## Builds on (most cited first, 30 of 52)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://mlsanity.com/paper/2205.01917/) (2022)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [Reproducible scaling laws for contrastive language-image learning](https://mlsanity.com/paper/2212.07143/) (2022)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [Striving for Simplicity: The All Convolutional Net](https://mlsanity.com/paper/1412.6806/) (2014)
- [Vision Transformers Need Registers](https://mlsanity.com/paper/2309.16588/) (2023)
- [Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned](https://mlsanity.com/paper/1905.09418/) (2019)
- [RISE: Randomized Input Sampling for Explanation of Black-box Models](https://mlsanity.com/paper/1806.07421/) (2018)
- [Quantifying Attention Flow in Transformers](https://mlsanity.com/paper/2005.00928/) (2020)
- [Transformer Interpretability Beyond Attention Visualization](https://mlsanity.com/paper/2012.09838/) (2020)
- [A Benchmark for Interpretability Methods in Deep Neural Networks](https://mlsanity.com/paper/1806.10758/) (2018)
- [Evaluating the visualization of what a Deep Neural Network has learned](https://mlsanity.com/paper/1509.06321/) (2015)
- [The (Un)reliability of saliency methods](https://mlsanity.com/paper/1711.00867/) (2017)

## Built on by (most cited first, 30 of 33)

- [B-cosification: Transforming Deep Neural Networks to be Inherently Interpretable](https://mlsanity.com/paper/2411.00715/) (2024)
- [Attention (as Discrete-Time Markov) Chains](https://mlsanity.com/paper/2507.17657/) (2025)
- [Visual Prompt Engineering for Vision Language Models in Radiology](https://mlsanity.com/paper/2408.15802/) (2024)
- [DAVE: Distribution-aware Attribution via ViT Gradient Decomposition](https://mlsanity.com/paper/2602.06613/) (2026)
- [Explaining Caption-Image Interactions in CLIP Models with Second-Order Attributions](https://mlsanity.com/paper/2408.14153/) (2024)
- [Symbolic Rule Extraction from Attention-Guided Sparse Representations in Vision Transformers](https://mlsanity.com/paper/2505.06745/) (2025)
- [LoRA-PT: Low-Rank Adapting UNETR for Hippocampus Segmentation Using Principal Tensor Singular Values and Vectors](https://mlsanity.com/paper/2407.11292/) (2024)
- [MaskInversion: Localized Embeddings via Optimization of Explainability Maps](https://mlsanity.com/paper/2407.20034/) (2024)
- [On the Explainability of Vision-Language Models in Art History](https://mlsanity.com/paper/2602.20853/) (2026)
- [VISTA: A Visual and Textual Attention Dataset for Interpreting Multimodal Models](https://mlsanity.com/paper/2410.04609/) (2024)
- [Explainability in AI Based Applications: A Framework for Comparing Different Techniques](https://mlsanity.com/paper/2410.20873/) (2024)
- [ULTra: Unveiling Latent Token Interpretability in Transformer-Based Understanding and Segmentation](https://mlsanity.com/paper/2411.12589/) (2024)
- [Find your Needle: Small Object Image Retrieval via Multi-Object Attention Optimization](https://mlsanity.com/paper/2503.07038/) (2025)
- [Focal Guidance: Unlocking Controllability from Semantic-Weak Layers in Video Diffusion Models](https://mlsanity.com/paper/2601.07287/) (2026)
- [DEX-AR: A Dynamic Explainability Method for Autoregressive Vision-Language Models](https://mlsanity.com/paper/2603.06302/) (2026)
- [Disentangling Hallucinations: Orthogonal Semantic Projection for Robust Interpretability](https://mlsanity.com/paper/2606.14758/) (2026)
- [Advancing Histopathology-Based Breast Cancer Diagnosis: Insights into Multi-Modality and Explainability](https://mlsanity.com/paper/2406.12897/) (2024)
- [Pose Matters: Evaluating Vision Transformers and CNNs for Human Action Recognition on Small COCO Subsets](https://mlsanity.com/paper/2506.11678/) (2025)
- [Leveraging Vision-Language Pre-training for Human Activity Recognition in Still Images](https://mlsanity.com/paper/2506.13458/) (2025)
- [Turning Adversaries into Allies: Reversing Typographic Attacks for Multimodal E-Commerce Product Retrieval](https://mlsanity.com/paper/2511.05325/) (2025)
- [Application of Graph Based Vision Transformers Architectures for Accurate Temperature Prediction in Fiber Specklegram Sensors](https://mlsanity.com/paper/2511.14792/) (2025)
- [TRANSPORTER: Transferring Visual Semantics from VLM Manifolds](https://mlsanity.com/paper/2511.18359/) (2025)
- [Learn to Rank: Visual Attribution by Learning Importance Ranking](https://mlsanity.com/paper/2604.05819/) (2026)
- [Decision-Aware Attention Propagation for Vision Transformer Explainability](https://mlsanity.com/paper/2604.18094/) (2026)
- [Interpretability Without Tradeoffs: Disentangling Polysemanticity At Equal Predictive Performance](https://mlsanity.com/paper/2605.31304/) (2026)
- [Does Aurora Encode Atmospheric Structure? Latent Regime Analysis and Attribution](https://mlsanity.com/paper/2606.26361/) (2026)
- [What Pixels Are Enough? SEAMS: Sufficiency Saliency via MSE-Preservation Soft-Masks](https://mlsanity.com/paper/2607.09164/) (2026)
- [Explaining AI-Image Detection: What the Heatmap Actually Shows](https://mlsanity.com/paper/2607.29581/) (2026)
- [SPICE: Simple Polysemantic Feature Interpretation via Clustering-based Explanation](https://mlsanity.com/paper/2609.13198/) (2026)
- [ResLRP: The Role of Residual Cancellation in Attribution Instability in Vision Transformers](https://mlsanity.com/paper/2609.17152/) (2026)

## Abstract (arXiv)

Vision Transformers (ViTs), with their ability to model long-range dependencies through self-attention mechanisms, have become a standard architecture in computer vision. However, the interpretability of these models remains a challenge. To address this, we propose LeGrad, an explainability method specifically designed for ViTs. LeGrad computes the gradient with respect to the attention maps of ViT layers, considering the gradient itself as the explainability signal. We aggregate the signal over all layers, combining the activations of the last as well as intermediate tokens to produce the merged explainability map. This makes LeGrad a conceptually simple and an easy-to-implement tool for enhancing the transparency of ViTs. We evaluate LeGrad in challenging segmentation, perturbation, and open-vocabulary settings, showcasing its versatility compared to other SotA explainability methods demonstrating its superior spatial fidelity and robustness to perturbations. A demo and the code is available at https://github.com/WalBouss/LeGrad.