mlsanityData to 14 Oct 2026

From Redundancy to Relevance: Information Flow in LVLMs Across Reasoning Tasks

arXiv 2406.06579 ↗First posted 4 Jun 2024cs.CL cs.AI cs.CVcited by 38

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Large Vision Language Models (LVLMs) achieve great performance on visual-language reasoning tasks, however, the black-box nature of LVLMs hinders in-depth research on the reasoning mechanism. As all images need to be converted into image tokens to fit the input format of large language models (LLMs) along with natural language prompts, sequential visual representation is essential to the performance of LVLMs, and the information flow analysis approach can be an effective tool for determining interactions between these representations. In this paper, we propose integrating attention analysis with LLaVA-CAM, concretely, attention scores highlight relevant regions during forward propagation, while LLaVA-CAM captures gradient changes through backward propagation, revealing key image features. By exploring the information flow from the perspective of visual representation contribution, we observe that it tends to converge in shallow layers but diversify in deeper layers. To validate our analysis, we conduct comprehensive experiments with truncation strategies across various LVLMs for visual question answering and image captioning tasks, and experimental results not only verify our hypothesis but also reveal a consistent pattern of information flow convergence in the corresponding layers, and the information flow cliff layer will be different due to different contexts. The paper's source code can be accessed from urlhttps://github.com/zhangbaijin/From-Redundancy-to-Relevance

Key facts. Cited by 38 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 18 of its 31 references that are in the map. First posted on arXiv 4 Jun 2024.

Builds on

18 of 31 references.

References · 18

  1. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  2. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  3. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  4. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  5. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  6. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  7. 2018Object Hallucination in Image Captioningcited by 669
  8. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
  9. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 615
  10. 2021Knowledge Neurons in Pretrained Transformerscited by 601
  11. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 365
  12. 2023Function Vectors in Large Language Modelscited by 308
6 more
  1. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 236
  2. 2023mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understandingcited by 168
  3. 2023ImageBind-LLM: Multi-modality Instruction Tuningcited by 142
  4. 2024LLaVA-Phi: Efficient Multi-Modal Assistant with Small Language Modelcited by 129
  5. 2023MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devicescited by 122
  6. 2019Smooth Grad-CAM++: An Enhanced Inference Level Visualization Technique for Deep Convolutional Neural Network Modelscited by 118

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 15; 2026: 18.

38 papers.

Sort:

By area:

  1. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 611
  2. 2024Seeing Clearly by Layer Two: Enhancing Attention Heads to Alleviate Hallucination in LVLMscited by 52
  3. 2025Backdoor Cleaning without External Guidance in MLLM Fine-tuningcited by 35
  4. 2025VFlowOpt: A Token Pruning Framework for LMMs with Visual Information Flow-Guided Optimizationcited by 22
  5. 2026Linear Mechanisms for Spatiotemporal Reasoning in Vision Language Modelscited by 14
  6. 2024Performance Gap in Entity Knowledge Extraction Across Modalities in Vision Language Modelscited by 12
  7. 2025VISTA: Enhancing Vision-Text Alignment in MLLMs via Cross-Modal Mutual Information Maximizationcited by 12
  8. 2025Position: Reasoning After Perception Means Reasoning Without Visioncited by 8
  9. 2025V-SEAM: Visual Semantic Editing and Attention Modulating for Causal Interpretability of Vision-Language Modelscited by 8
  10. 2025Revealing Perception and Generation Dynamics in LVLMs: Mitigating Hallucinations via Validated Dominance Correctioncited by 7
  11. 2025SurgXBench: Explainable Vision-Language Model Benchmark for Surgerycited by 6
  12. 2026Context Tokens are Anchors: Understanding the Repetition Curse in dMLLMs from an Information Flow Perspectivecited by 6
26 more
  1. 2025AdaToken-3D: Dynamic Spatial Gating for Efficient 3D Large Multimodal-Models Reasoningcited by 5
  2. 2025EchoVLM: Dynamic Mixture-of-Experts Vision-Language Model for Universal Ultrasound Intelligencecited by 5
  3. 2025Seeing It or Not? Interpretable Vision-aware Latent Steering to Mitigate Object Hallucinationscited by 3
  4. 2026Visual Enhanced Depth Scaling for Multimodal Latent Reasoningcited by 3
  5. 2026Aligning What Vision-Language Models See and Perceive with Adaptive Information Flowcited by 3
  6. 2025Reading Images Like Texts: Sequential Image Understanding in Vision-Language Modelscited by 2
  7. 2025Unleashing the Intrinsic Visual Representation Capability of Multimodal Large Language Modelscited by 2
  8. 2026Where Not to Learn: Prior-Aligned Training with Subset-based Attribution Constraintscited by 2
  9. 2026SAGE: Sink-Aware Grounded Decoding for Multimodal Hallucination Mitigationcited by 2
  10. 2026A Single Layer to Explain Them All:Understanding Massive Activations in Large Language Modelscited by 2
  11. 2024Enhancing Multimodal Large Language Models Complex Reason via Similarity Computationcited by 1
  12. 2025Interpreting Social Bias in LVLMs via Information Flow Analysis and Multi-Round Dialogue Evaluationcited by 1
  13. 2026ART: Attention Replacement Technique to Improve Factuality in LLMscited by 1
  14. 2026Diffusion-CAM: Faithful Visual Explanations for dMLLMscited by 1
  15. 2026Measuring Cross-Modal Synergy: A Benchmark for VLM Explainabilitycited by 1
  16. 2026When Attention Collapses: Stage-Aware Visual Token Pruning from Structure to Semanticscited by 1
  17. 2026Where to Place the Query? Unveiling and Mitigating Positional Bias in In-Context Learning for Diffusion LLMs via Decoding Dynamicscited by 1
  18. 2024Toward Modality Gap: Vision Prototype Learning for Weakly-supervised Semantic Segmentation with CLIPcited by 0
  19. 2025Interpretable graph-based models on multimodal biomedical data integration: A technical review and benchmarkingcited by 0
  20. 2025Malicious Image Analysis via Vision-Language Segmentation Fusion: Detection, Element, and Location in One-shotcited by 0
  21. 2026Visualising Information Flow in Word Embeddings with Diffusion Tensor Imagingcited by 0
  22. 2026Tinted Frames: Question Framing Blinds Vision-Language Modelscited by 0
  23. 2026A Good Initialization is All You Need for Faithful Visual Attributioncited by 0
  24. 2026Beyond Language Priors: Diagnosing and Fixing Visual-Origin Hallucinations in Multimodal LLMcited by 0
  25. 2026From Vision to Language: Investigating Causal Information Flow in Multimodal Decision-Makingcited by 0
  26. 2026Beyond Entropy: Self-Diagnostic Multi-Role Token Optimization for Video Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Where do Large Vision-Language Models Look at when Answering Questions?
  2. 2024Cross-modal Information Flow in Multimodal Large Language Models
  3. 2025Lifting the Veil on Visual Information Flow in MLLMs: Unlocking Pathways to Faster Inference
  4. 2025Understanding Language Prior of LVLMs by Contrasting Chain-of-Embedding
  5. 2025Interpreting Attention Heads for Image-to-Text Information Flow in Large Vision-Language Models
  6. 2024Cross-Lingual Text-Rich Visual Comprehension: An Information Theory Perspective
  7. 2024A-VL: Adaptive Attention for Large Vision-Language Models
  8. 2025How Multimodal LLMs Solve Image Tasks: A Lens on Visual Grounding, Task Reasoning, and Answer Decoding