# Where MLLMs Attend and What They Rely On: Explaining Autoregressive Token Generation

arXiv: https://arxiv.org/abs/2509.22496 · HTML: https://mlsanity.com/paper/2509.22496/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 30 of its 42 references that are in the map.
- First posted on arXiv 26 Sep 2025.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)

## Builds on (most cited first, 30 of 30)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency](https://mlsanity.com/paper/2508.18265/) (2025)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [RISE: Randomized Input Sampling for Explanation of Black-box Models](https://mlsanity.com/paper/1806.07421/) (2018)
- [Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs](https://mlsanity.com/paper/2401.06209/) (2024)
- [End-to-end Autonomous Driving: Challenges and Frontiers](https://mlsanity.com/paper/2306.16927/) (2023)
- [Top-down Neural Attention by Excitation Backprop](https://mlsanity.com/paper/1608.00507/) (2016)
- [MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMs](https://mlsanity.com/paper/2502.17422/) (2025)
- [Smooth Grad-CAM++: An Enhanced Inference Level Visualization Technique for Deep Convolutional Neural Network Models](https://mlsanity.com/paper/1908.01224/) (2019)
- [BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learning](https://mlsanity.com/paper/2311.12075/) (2023)
- [Interpreting and Editing Vision-Language Representations to Mitigate Hallucinations](https://mlsanity.com/paper/2410.02762/) (2024)
- [Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2412.02104/) (2024)
- [Revisiting Backdoor Attacks against Large Vision-Language Models from Domain Shift](https://mlsanity.com/paper/2406.18844/) (2024)
- [Visual Large Language Models for Generalized and Specialized Applications](https://mlsanity.com/paper/2501.02765/) (2025)
- [A Survey on Mechanistic Interpretability for Multi-Modal Foundation Models](https://mlsanity.com/paper/2502.17516/) (2025)
- [Token Activation Map to Visually Explain Multimodal LLMs](https://mlsanity.com/paper/2506.23270/) (2025)
- [Guaranteed Non-convex Optimization: Submodular Maximization over Continuous Domains](https://mlsanity.com/paper/1606.05615/) (2016)
- [A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks](https://mlsanity.com/paper/2411.06284/) (2024)
- [Interpreting Object-level Foundation Models via Visual Precision Search](https://mlsanity.com/paper/2411.16198/) (2024)
- [Where do Large Vision-Language Models Look at when Answering Questions?](https://mlsanity.com/paper/2503.13891/) (2025)
- [SafeMobile: Chain-level Jailbreak Detection and Automated Evaluation for Multimodal Mobile Agents](https://mlsanity.com/paper/2507.00841/) (2025)
- [E2AT: Multimodal Jailbreak Defense via Dynamic Joint Optimization for Multimodal Large Language Models](https://mlsanity.com/paper/2503.04833/) (2025)
- [DeFacto: Counterfactual Thinking with Images for Enforcing Evidence-Grounded and Faithful Reasoning](https://mlsanity.com/paper/2509.20912/) (2025)
- [RePOPE: Impact of Annotation Errors on the POPE Benchmark](https://mlsanity.com/paper/2504.15707/) (2025)
- [Seeing It or Not? Interpretable Vision-aware Latent Steering to Mitigate Object Hallucinations](https://mlsanity.com/paper/2505.17812/) (2025)

## Built on by (most cited first, 16 of 16)

- [EgoVITA: Learning to Plan and Verify for Egocentric Video Reasoning](https://mlsanity.com/paper/2511.18242/) (2025)
- [Where Not to Learn: Prior-Aligned Training with Subset-based Attribution Constraints](https://mlsanity.com/paper/2602.07008/) (2026)
- [Can Attribution Predict Risk? From Multi-View Attribution to Planning Risk Signals in End-to-End Autonomous Driving](https://mlsanity.com/paper/2605.06264/) (2026)
- [Did Models Learn Sufficiently? Attribution-Guided Training via Subset-Selected Counterfactual Augmentation](https://mlsanity.com/paper/2511.12100/) (2025)
- [Measuring Cross-Modal Synergy: A Benchmark for VLM Explainability](https://mlsanity.com/paper/2605.22168/) (2026)
- [PhaseWin: An Efficient Search Algorithm for Faithful Visual Attribution](https://mlsanity.com/paper/2606.18008/) (2026)
- [Which Modality Decides? Counterfactual Modality Attribution for Multimodal LLMs](https://mlsanity.com/paper/2608.00076/) (2026)
- [Towards Reliable Medical LLMs: Benchmarking and Enhancing Confidence Estimation of Large Language Models in Medical Consultation](https://mlsanity.com/paper/2601.15645/) (2026)
- [LatentGeo: Learnable Auxiliary Constructions in Latent Space for Multimodal Geometric Reasoning](https://mlsanity.com/paper/2603.12166/) (2026)
- [PGT: Procedurally Generated Tasks for improving visual grounding in MLLMs](https://mlsanity.com/paper/2605.23883/) (2026)
- [Attending to Multimodal Generation One Token at a Time](https://mlsanity.com/paper/2607.03738/) (2026)
- [A Good Initialization is All You Need for Faithful Visual Attribution](https://mlsanity.com/paper/2607.06726/) (2026)
- [Consistent Evidence, Robust Recognition: Faithful Attribution Regularization under Geometric Transformations](https://mlsanity.com/paper/2607.23835/) (2026)
- [IntentQA: Intent Question Answering in Videos by Cognitive Context Reasoning](https://mlsanity.com/paper/2608.23330/) (2026)
- [What do VLM-Based Vision-Language Navigation Models Rely on: Interpreting and Steering Policy Behavior](https://mlsanity.com/paper/2609.24576/) (2026)
- [OmniConfess: Eliciting Token Confessions to Mitigate Omni-Modal Hallucination](https://mlsanity.com/paper/2610.02999/) (2026)

## Abstract (arXiv)

Multimodal large language models (MLLMs) have demonstrated remarkable capabilities in aligning visual inputs with natural language outputs. Yet, the extent to which generated tokens depend on visual modalities remains poorly understood, limiting interpretability and reliability. In this work, we present EAGLE, a lightweight black-box framework for explaining autoregressive token generation in MLLMs. EAGLE attributes any selected tokens to compact perceptual regions while quantifying the relative influence of language priors and perceptual evidence. The framework introduces an objective function that unifies sufficiency (insight score) and indispensability (necessity score), optimized via greedy search over sparsified image regions for faithful and efficient attribution. Beyond spatial attribution, EAGLE performs modality-aware analysis that disentangles what tokens rely on, providing fine-grained interpretability of model decisions. Extensive experiments across open-source MLLMs show that EAGLE consistently outperforms existing methods in faithfulness, localization, and hallucination diagnosis, while requiring substantially less GPU memory. These results highlight its effectiveness and practicality for advancing the interpretability of MLLMs.