# GMAR: Gradient-Driven Multi-Head Attention Rollout for Vision Transformer Interpretability

arXiv: https://arxiv.org/abs/2504.19414 · HTML: https://mlsanity.com/paper/2504.19414/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 27 references that are in the map.
- First posted on arXiv 28 Apr 2025.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned](https://mlsanity.com/paper/1905.09418/) (2019)
- [Quantifying Attention Flow in Transformers](https://mlsanity.com/paper/2005.00928/) (2020)
- [Transformer Interpretability Beyond Attention Visualization](https://mlsanity.com/paper/2012.09838/) (2020)
- [Visual Transformers: Token-based Image Representation and Processing for Computer Vision](https://mlsanity.com/paper/2006.03677/) (2020)
- [XAI for Transformers: Better Explanations through Conservative Propagation](https://mlsanity.com/paper/2202.07304/) (2022)
- [MoH: Multi-Head Attention as Mixture-of-Head Attention](https://mlsanity.com/paper/2410.11842/) (2024)
- [How Does Attention Work in Vision Transformers? A Visual Analytics Attempt](https://mlsanity.com/paper/2303.13731/) (2023)
- [ViT-CX: Causal Explanation of Vision Transformers](https://mlsanity.com/paper/2211.03064/) (2022)
- [FovEx: Human-Inspired Explanations for Vision Transformers and Convolutional Neural Networks](https://mlsanity.com/paper/2408.02123/) (2024)

## Built on by (most cited first, 11 of 11)

- [How Vision Becomes Language: A Layer-wise Information-Theoretic Analysis of Multimodal Reasoning](https://mlsanity.com/paper/2602.15580/) (2026)
- [FairViT-GAN: A Hybrid Vision Transformer with Adversarial Debiasing for Fair and Explainable Facial Beauty Prediction](https://mlsanity.com/paper/2509.23859/) (2025)
- [Central Dogma Transformer: Towards Mechanism-Oriented AI for Cellular Understanding](https://mlsanity.com/paper/2601.01089/) (2026)
- [Modeling Day-Long ECG Signals to Predict Heart Failure Risk with Explainable AI](https://mlsanity.com/paper/2601.00014/) (2025)
- [Decision-Aware Attention Propagation for Vision Transformer Explainability](https://mlsanity.com/paper/2604.18094/) (2026)
- [Revealing the Gap in Human and VLM Scene Perception through Counterfactual Semantic Saliency](https://mlsanity.com/paper/2605.13047/) (2026)
- [KANEx: Translating Kolmogorov-Arnold Networks' Interpretability to Medical Explainability](https://mlsanity.com/paper/2607.24730/) (2026)
- [Faithful Faithfulness Evaluations: Challenges & Pitfalls Learned from a Breast MRI Case Study](https://mlsanity.com/paper/2609.25978/) (2026)
- [Scaling Attention Head Analysis via Gradient-Based Attribution in Context-Aware Machine Translation](https://mlsanity.com/paper/2609.28117/) (2026)
- [Data-Adaptive Mahalanobis Metric Learning for Cross-Head Attention in Transformers](https://mlsanity.com/paper/2610.13261/) (2026)
- [When, Where, and Why: Hierarchical Causal Explainability for Video Face Understanding with MLLMs](https://mlsanity.com/paper/2610.17791/) (2026)

## Abstract (arXiv)

The Vision Transformer (ViT) has made significant advancements in computer vision, utilizing self-attention mechanisms to achieve state-of-the-art performance across various tasks, including image classification, object detection, and segmentation. Its architectural flexibility and capabilities have made it a preferred choice among researchers and practitioners. However, the intricate multi-head attention mechanism of ViT presents significant challenges to interpretability, as the underlying prediction process remains opaque. A critical limitation arises from an observation commonly noted in transformer architectures: "Not all attention heads are equally meaningful." Overlooking the relative importance of specific heads highlights the limitations of existing interpretability methods. To address these challenges, we introduce Gradient-Driven Multi-Head Attention Rollout (GMAR), a novel method that quantifies the importance of each attention head using gradient-based scores. These scores are normalized to derive a weighted aggregate attention score, effectively capturing the relative contributions of individual heads. GMAR clarifies the role of each head in the prediction process, enabling more precise interpretability at the head level. Experimental results demonstrate that GMAR consistently outperforms traditional attention rollout techniques. This work provides a practical contribution to transformer-based architectures, establishing a robust framework for enhancing the interpretability of Vision Transformer models.