mlsanityData to 14 Oct 2026

Where do Large Vision-Language Models Look at when Answering Questions?

arXiv 2503.13891 ↗First posted 18 Mar 2025cs.CV cs.CLcited by 15

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Large Vision-Language Models (LVLMs) have shown promising performance in vision-language understanding and reasoning tasks. However, their visual understanding behaviors remain underexplored. A fundamental question arises: to what extent do LVLMs rely on visual input, and which image regions contribute to their responses? It is non-trivial to interpret the free-form generation of LVLMs due to their complicated visual architecture (e.g., multiple encoders and multi-resolution) and variable-length outputs. In this paper, we extend existing heatmap visualization methods (e.g., iGOS++) to support LVLMs for open-ended visual question answering. We propose a method to select visually relevant tokens that reflect the relevance between generated answers and input image. Furthermore, we conduct a comprehensive analysis of state-of-the-art LVLMs on benchmarks designed to require visual information to answer. Our findings offer several insights into LVLM behavior, including the relationship between focus region and answer correctness, differences in visual attention across architectures, and the impact of LLM scale on visual understanding. The code and data are available at https://github.com/bytedance/LVLM_Interpretation.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 41 of its 54 references that are in the map. First posted on arXiv 18 Mar 2025.

Builds on

41 of 54 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?29 Mar
  2. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models27 Mar

Other references · 39

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  10. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  11. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  12. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
27 more
  1. 2015VQA: Visual Question Answeringcited by 2,506
  2. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,371
  3. 2013Visualizing and Understanding Convolutional Networkscited by 2,290
  4. 2024Qwen2 Technical Reportcited by 2,120
  5. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  6. 2018RISE: Randomized Input Sampling for Explanation of Black-box Modelscited by 838
  7. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  8. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  9. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 642
  10. 2020Transformer Interpretability Beyond Attention Visualizationcited by 627
  11. 2017Interpretable Explanations of Black Boxes by Meaningful Perturbationcited by 612
  12. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 478
  13. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 445
  14. 2016Top-down Neural Attention by Excitation Backpropcited by 321
  15. 2021Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformerscited by 320
  16. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Modelscited by 316
  17. 2023How Language Model Hallucinations Can Snowballcited by 311
  18. 2017Real Time Image Saliency for Black Box Classifierscited by 246
  19. 2023Hallucination Augmented Contrastive Learning for Multimodal Large Language Modelcited by 121
  20. 2016Human Attention in Visual Question Answering: Do Humans and Deep Networks Look at the Same Regions?cited by 111
  21. 2019Visualizing Deep Networks by Optimizing with Integrated Gradientscited by 59
  22. 2015On Graduated Optimization for Stochastic Non-Convex Problemscited by 34
  23. 2021Multimodal Integration of Human-Like Attention in Visual Question Answeringcited by 20
  24. 2020iGOS++: Integrated Gradient Optimized Saliency by Bilateral Perturbationscited by 19
  25. 2023Visual Explanations via Iterated Integrated Attributionscited by 19
  26. 2022Comparing the Decision-Making Mechanisms by Transformers and CNNs via Explanation Methodscited by 12
  27. 2024Explaining Multi-modal Large Language Models by Analyzing their Vision Perceptioncited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 9.

15 papers.

Sort:

By area:

  1. 2025Where MLLMs Attend and What They Rely On: Explaining Autoregressive Token Generationcited by 16
  2. 2025V-SEAM: Visual Semantic Editing and Attention Modulating for Causal Interpretability of Vision-Language Modelscited by 8
  3. 2025LaSM: Layer-wise Scaling Mechanism for Defending Pop-up Attack on GUI Agentscited by 6
  4. 2025Seeing It or Not? Interpretable Vision-aware Latent Steering to Mitigate Object Hallucinationscited by 3
  5. 2025SMA: Who Said That? Auditing Membership Leakage in Semi-Black-box RAG Controllingcited by 3
  6. 2026Focusing Where Vision Matters: Selective Training for Large Vision Language Models via Visual Information Gaincited by 3
  7. 2026From Actions to Understanding: Conformal Interpretability of Temporal Concepts in LLM Agentscited by 3
  8. 2026Where Not to Learn: Prior-Aligned Training with Subset-based Attribution Constraintscited by 2
  9. 2026TraceVision: Trajectory-Aware Vision-Language Model for Human-Like Spatial Understandingcited by 2
  10. 2025Towards Accurate UAV Image Perception: Guiding Vision-Language Models with Stronger Task Promptscited by 1
  11. 2026Measuring Cross-Modal Synergy: A Benchmark for VLM Explainabilitycited by 1
  12. 2026PhaseWin: An Efficient Search Algorithm for Faithful Visual Attributioncited by 1
3 more
  1. 2026A Good Initialization is All You Need for Faithful Visual Attributioncited by 0
  2. 2026Clustering and Token Denoising for Faster and More Robust VLMscited by 0
  3. 2026AnswerMap: Faithful Spatial Interpretability of VLMs from Answer Posteriorscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challenges
  2. 2024FiVL: A Framework for Improved Vision-Language Alignment through the Lens of Training, Evaluation and Explainability
  3. 2025Elevating Visual Question Answering through Implicitly Learned Reasoning Pathways in LVLMs
  4. 2026Understanding the Fine-Grained Knowledge Capabilities of Vision-Language Models
  5. 2025MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMs
  6. 2024Unveiling the Tapestry of Consistency in Large Vision-Language Models
  7. 2024Understanding Multimodal LLMs: the Mechanistic Interpretability of Llava in Visual Question Answering
  8. 2024From Redundancy to Relevance: Information Flow in LVLMs Across Reasoning Tasks