# Token Activation Map to Visually Explain Multimodal LLMs

arXiv: https://arxiv.org/abs/2506.23270 · HTML: https://mlsanity.com/paper/2506.23270/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 46 of its 57 references that are in the map.
- First posted on arXiv 29 Jun 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Visual token pruning and efficiency](https://mlsanity.com/mllm/visual-token-pruning-and-efficiency/)

## Builds on (most cited first, 30 of 46)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://mlsanity.com/paper/2412.05271/) (2024)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge](https://mlsanity.com/paper/1906.00067/) (2019)
- [Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networks](https://mlsanity.com/paper/1710.11063/) (2017)
- [Vision Transformers Need Registers](https://mlsanity.com/paper/2309.16588/) (2023)
- [Quantifying Attention Flow in Transformers](https://mlsanity.com/paper/2005.00928/) (2020)
- [Transformer Interpretability Beyond Attention Visualization](https://mlsanity.com/paper/2012.09838/) (2020)
- [GLaMM: Pixel Grounding Large Multimodal Model](https://mlsanity.com/paper/2311.03356/) (2023)
- [Unmasking Clever Hans Predictors and Assessing What Machines Really Learn](https://mlsanity.com/paper/1902.10178/) (2019)
- [Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers](https://mlsanity.com/paper/2103.15679/) (2021)
- [Text2LIVE: Text-Driven Layered Image and Video Editing](https://mlsanity.com/paper/2204.02491/) (2022)
- [A Survey on Multimodal Large Language Models for Autonomous Driving](https://mlsanity.com/paper/2311.12320/) (2023)
- [STAR: A Benchmark for Situated Reasoning in Real-World Videos](https://mlsanity.com/paper/2405.09711/) (2024)

## Built on by (most cited first, 30 of 32)

- [Patch-as-Decodable-Token: Towards Unified Multi-Modal Vision Tasks in MLLMs](https://mlsanity.com/paper/2510.01954/) (2025)
- [Where MLLMs Attend and What They Rely On: Explaining Autoregressive Token Generation](https://mlsanity.com/paper/2509.22496/) (2025)
- [Evidence Recomposition and Predictive Context Residualization for Visual Attribution in Multimodal Large Language Models](https://mlsanity.com/paper/2509.22415/) (2025)
- [Visual Enhanced Depth Scaling for Multimodal Latent Reasoning](https://mlsanity.com/paper/2604.10500/) (2026)
- [Reading Images Like Texts: Sequential Image Understanding in Vision-Language Models](https://mlsanity.com/paper/2509.19191/) (2025)
- [Decomposed Attention Fusion in MLLMs for Training-Free Video Reasoning Segmentation](https://mlsanity.com/paper/2510.19592/) (2025)
- [Where Not to Learn: Prior-Aligned Training with Subset-based Attribution Constraints](https://mlsanity.com/paper/2602.07008/) (2026)
- [SPD-Faith Bench: Diagnosing and Improving Faithfulness in Chain-of-Thought for Multimodal Large Language Models](https://mlsanity.com/paper/2602.07833/) (2026)
- [One Token, Two Fates: A Unified Framework via Vision Token Manipulation Against MLLMs Hallucination](https://mlsanity.com/paper/2603.10360/) (2026)
- [Saliency-R1: Enforcing Interpretable and Faithful Vision-language Reasoning via Saliency-map Alignment Reward](https://mlsanity.com/paper/2604.04500/) (2026)
- [DEX-AR: A Dynamic Explainability Method for Autoregressive Vision-Language Models](https://mlsanity.com/paper/2603.06302/) (2026)
- [Diffusion-CAM: Faithful Visual Explanations for dMLLMs](https://mlsanity.com/paper/2604.11005/) (2026)
- [MACS: Modality-Aware Capacity Scaling for Efficient Multimodal MoE Inference](https://mlsanity.com/paper/2605.05225/) (2026)
- [PaperFit: Vision-in-the-Loop Typesetting Optimization for Scientific Documents](https://mlsanity.com/paper/2605.10341/) (2026)
- [Measuring Cross-Modal Synergy: A Benchmark for VLM Explainability](https://mlsanity.com/paper/2605.22168/) (2026)
- [PhaseWin: An Efficient Search Algorithm for Faithful Visual Attribution](https://mlsanity.com/paper/2606.18008/) (2026)
- [Which Modality Decides? Counterfactual Modality Attribution for Multimodal LLMs](https://mlsanity.com/paper/2608.00076/) (2026)
- [VERHallu: Evaluating and Mitigating Event Relation Hallucination in Video Large Language Models](https://mlsanity.com/paper/2601.10010/) (2026)
- [DenseMLLM: Standard Multimodal LLMs for Dense Prediction](https://mlsanity.com/paper/2602.14134/) (2026)
- [Visual Attention Drifts,but Anchors Hold:Mitigating Hallucination in Multimodal Large Language Models via Cross-Layer Visual Anchors](https://mlsanity.com/paper/2603.25088/) (2026)
- [InstAP: Instance-Aware Vision-Language Pre-Train for Spatial-Temporal Understanding](https://mlsanity.com/paper/2604.08337/) (2026)
- [What-Where Transformer: A Slot-Centric Visual Backbone for Concurrent Representation and Localization](https://mlsanity.com/paper/2605.12021/) (2026)
- [Listening makes Vision Clear for VLMs](https://mlsanity.com/paper/2606.23763/) (2026)
- [Understanding How MLLMs Describe Artworks Using Token Activation Maps](https://mlsanity.com/paper/2606.27947/) (2026)
- [A Good Initialization is All You Need for Faithful Visual Attribution](https://mlsanity.com/paper/2607.06726/) (2026)
- [Grad-CAM for Vision Transformers: A Systematic Taxonomy and Audit of Methodological Ambiguity in Explainable AI](https://mlsanity.com/paper/2608.05258/) (2026)
- [Zero-MELO: Test-Time Evidence Calibration with Multimodal LLMs for Zero-Shot Micro-Gesture Recognition](https://mlsanity.com/paper/2608.14854/) (2026)
- [What Did the MLLM Hear? Token-Level Spectro-Temporal Grounding for Audio MLLM Explainability](https://mlsanity.com/paper/2609.12663/) (2026)
- [What do VLM-Based Vision-Language Navigation Models Rely on: Interpreting and Steering Policy Behavior](https://mlsanity.com/paper/2609.24576/) (2026)
- [AnswerMap: Faithful Spatial Interpretability of VLMs from Answer Posteriors](https://mlsanity.com/paper/2609.35247/) (2026)

## Abstract (arXiv)

Multimodal large language models (MLLMs) are broadly empowering various fields. Despite their advancements, the explainability of MLLMs remains less explored, hindering deeper understanding, model credibility, and effective visualization. Unlike conventional vision models (e.g., CNNs, ViTs, CLIP) that produce a single output, MLLMs generate sequences of tokens progressively, where each generated token depends on the previous context. Therefore, earlier context tokens can introduce redundant activations that interfere with the explanation of later tokens beyond their original information. Existing studies often overlook this issue, but our observations reveal that these redundant correlations can significantly hurt the reliability of explanations. To address this, we propose an estimated causal inference method to mitigate the interference of context to achieve high-quality MLLM explanation, with a novel rank Gaussian filter to further reduce activation noises. We term this method Token Activation Map (TAM) to highlight the consideration of interactions between tokens. TAM also indicates that it excels at explaining multiple tokens of MLLM, which is different from the Class Activation Map (CAM) for a single prediction. Our TAM method significantly outperforms existing SoTA methods, showcasing high-quality visualization results that can be utilized for various scenarios, such as object localization, failure case analysis, video visualization, MLLMs visual comparison, and model understanding (e.g., color, shape, action, location, visual reasoning, multi-turn conversation, etc). The code is available atgithub.com/xmed-lab/TAM.