mlsanityData to 14 Oct 2026

Towards Interpreting Visual Information Processing in Vision-Language Models

arXiv 2410.07149 ↗First posted 9 Oct 2024cs.CV cs.LGcited by 122

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-Language Models (VLMs) are powerful tools for processing and understanding text and images. We study the processing of visual tokens in the language model component of LLaVA, a prominent VLM. Our approach focuses on analyzing the localization of object information, the evolution of visual token representations across layers, and the mechanism of integrating visual information for predictions. Through ablation studies, we demonstrated that object identification accuracy drops by over 70% when object-specific tokens are removed. We observed that visual token representations become increasingly interpretable in the vocabulary space across layers, suggesting an alignment with textual tokens corresponding to image content. Finally, we found that the model extracts object information from these refined representations at the last token position for prediction, mirroring the process in text-only language models for factual association tasks. These findings provide crucial insights into how VLMs process and integrate visual information, bridging the gap between our understanding of language and vision models, and paving the way for more interpretable and controllable multimodal systems.

Key facts. Cited by 122 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 37 of its 52 references that are in the map. First posted on arXiv 9 Oct 2024.

Builds on

37 of 52 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Understanding Information Storage and Transfer in Multi-modal Large Language Models6 Jun
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr

Other references · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  10. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  11. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  12. 2017Axiomatic Attribution for Deep Networkscited by 3,384
23 more
  1. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  2. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  3. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  4. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  5. 2023Vision Transformers Need Registerscited by 871
  6. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  7. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 532
  8. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 528
  9. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 435
  10. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 281
  11. 2023Interpreting CLIP's Image Representation via Text-Based Decompositioncited by 196
  12. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMscited by 179
  13. 2022Linearly Mapping from Image to Text Spacecited by 142
  14. 2023Hallucination Augmented Contrastive Learning for Multimodal Large Language Modelcited by 121
  15. 2024Interpreting and Editing Vision-Language Representations to Mitigate Hallucinationscited by 104
  16. 2024Why are Visually-Grounded Language Models Bad at Image Classification?cited by 102
  17. 2024Discover-then-Name: Task-Agnostic Concept Bottlenecks via Automated Concept Discoverycited by 91
  18. 2023A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysiscited by 89
  19. 2023Towards Vision-Language Mechanistic Interpretability: A Causal Tracing Tool for BLIPcited by 62
  20. 2023Multimodal Neurons in Pretrained Text-Only Transformerscited by 49
  21. 2024Interpreting the Second-Order Effects of Neurons in CLIPcited by 36
  22. 2023Analyzing Vision Transformers for Image Classification in Class Embedding Spacecited by 26
  23. 2024Dissecting Query-Key Interaction in Vision Transformerscited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 44; 2026: 75.

122 papers.

Sort:

By area:

  1. 2025See What You Are Told: Visual Attention Sink in Large Multimodal Modelscited by 132
  2. 2024What's in the Image? A Deep-Dive into the Vision of Vision Language Modelscited by 68
  3. 2024Cross-modal Information Flow in Multimodal Large Language Modelscited by 67
  4. 2025VAGEN: Reinforcing World Model Reasoning for Multi-Turn VLM Agentscited by 62
  5. 2025InfiniPot-V: Memory-Constrained KV Cache Compression for Streaming Video Understandingcited by 58
  6. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  7. 2025Don't Just Chase "Highlighted Tokens" in MLLMs: Revisiting Visual Holistic Context Retentioncited by 47
  8. 2025Visual Representation Alignment for Multimodal Large Language Modelscited by 41
  9. 2025Beyond Semantics: Rediscovering Spatial Awareness in Vision-Language Modelscited by 37
  10. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 36
  11. 2025Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMscited by 35
  12. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 27
38 more
  1. 2025How Visual Representations Map to Language Feature Space in Multimodal LLMscited by 25
  2. 2025How Do Vision-Language Models Process Conflicting Information Across Modalities?cited by 23
  3. 2025FOCUS: Internal MLLM Representations for Efficient Fine-Grained Visual Question Answeringcited by 20
  4. 2025AudioLens: A Closer Look at Auditory Attribute Perception of Large Audio-Language Modelscited by 18
  5. 2025Vision Function Layer in Multimodal LLMscited by 17
  6. 2025VisiPruner: Decoding Discontinuous Cross-Modal Dynamics for Efficient Multimodal LLMscited by 17
  7. 2025Head Pursuit: Probing Attention Specialization in Multimodal Transformerscited by 17
  8. 2025Hallucinatory Image Tokens: A Training-free EAZY Approach on Detecting and Mitigating Object Hallucinations in LVLMscited by 15
  9. 2025Visual Representations inside the Language Modelcited by 15
  10. 2025MVI-Bench: A Comprehensive Benchmark for Evaluating Robustness to Misleading Visual Inputs in LVLMscited by 15
  11. 2026LatentLens: Revealing Highly Interpretable Visual Tokens in LLMscited by 15
  12. 2025Understanding Language Prior of LVLMs by Contrasting Chain-of-Embeddingcited by 14
  13. 2025CompoDistill: Attention Distillation for Compositional Reasoning in Multimodal LLMscited by 14
  14. 2026Linear Mechanisms for Spatiotemporal Reasoning in Vision Language Modelscited by 14
  15. 2025Visual symbolic mechanisms: Emergent symbol processing in vision language modelscited by 13
  16. 2025How Multimodal LLMs Solve Image Tasks: A Lens on Visual Grounding, Task Reasoning, and Answer Decodingcited by 13
  17. 2025Visual Multi-Agent System: Mitigating Hallucination Snowballing via Visual Flowcited by 13
  18. 2026VLMs Need Words: Vision Language Models Ignore Visual Detail In Favor of Semantic Anchorscited by 13
  19. 2025Hierarchical Cross-modal Prompt Learning for Vision-Language Modelscited by 12
  20. 2026Arbitration Failure, Not Perceptual Blindness: How Vision-Language Models Resolve Visual-Linguistic Conflictscited by 11
  21. 2025Map the Flow: Revealing Hidden Pathways of Information in VideoLLMscited by 10
  22. 2025Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Modelscited by 9
  23. 2026Do Audio-Visual Large Language Models Really See and Hear?cited by 9
  24. 2025Too Late to Recall: Explaining the Two-Hop Problem in Multimodal Knowledge Retrievalcited by 7
  25. 2026Fine-Grained Activation Steering: Steering Less, Achieving Morecited by 7
  26. 2026What Do Visual Tokens Really Encode? Uncovering Sparsity and Redundancy in Multimodal Large Language Modelscited by 7
  27. 2026Dual-Pathway Circuits of Object Hallucination in Vision-Language Modelscited by 7
  28. 2025Rethinking Explainability in the Era of Multimodal AIcited by 6
  29. 2025[De|Re]constructing VLMs' Reasoning in Countingcited by 6
  30. 2025Interpretable and Steerable Concept Bottleneck Sparse Autoencoderscited by 6
  31. 2026Instruction Anchor: Dissecting the Mechanistic Dynamics of Modality Arbitrationcited by 6
  32. 2026Interpretable Debiasing of Vision-Language Models for Social Fairnesscited by 6
  33. 2025The Point, the Vision and the Text: Does Point Cloud Boost Spatial Reasoning of Large Language Models? A Bias-Controlled Studycited by 5
  34. 2025Seeing the Threat: Vulnerabilities in Vision-Language Models to Adversarial Attackcited by 5
  35. 2025Investigating Mechanisms for In-Context Vision Language Bindingcited by 5
  36. 2026Test-Time Attention Purification for Backdoored Large Vision Language Modelscited by 5
  37. 2026Probing Cross-modal Information Hubs in Audio-Visual LLMscited by 5
  38. 2026Gaze Heads: How VLMs Look at What They Describecited by 5

50 most cited of 122 All 122 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Do Vision Language Models Need to Process Image Tokens?
  2. 2026Aligning What Vision-Language Models See and Perceive with Adaptive Information Flow
  3. 2024The Narrow Gate: Localized Image-Text Communication in Native Multimodal Models
  4. 2025Hidden in plain sight: VLMs overlook their visual representations
  5. 2025VScan: Rethinking Visual Token Reduction for Efficient Large Vision-Language Models
  6. 2023Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding
  7. 2024Unveiling Visual Perception in Language Models: An Attention Head Analysis Approach
  8. 2024Treat Visual Tokens as Text? But Your MLLM Only Needs Fewer Efforts to See