# From Redundancy to Relevance: Information Flow in LVLMs Across Reasoning Tasks

arXiv: https://arxiv.org/abs/2406.06579 · HTML: https://mlsanity.com/paper/2406.06579/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 18 of its 31 references that are in the map.
- First posted on arXiv 4 Jun 2024.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 18 of 18)

- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [Evaluating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2305.10355/) (2023)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019)
- [Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic](https://mlsanity.com/paper/2306.15195/) (2023)
- [Object Hallucination in Image Captioning](https://mlsanity.com/paper/1809.02156/) (2018)
- [MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning](https://mlsanity.com/paper/2310.09478/) (2023)
- [NExT-GPT: Any-to-Any Multimodal LLM](https://mlsanity.com/paper/2309.05519/) (2023)
- [Knowledge Neurons in Pretrained Transformers](https://mlsanity.com/paper/2104.08696/) (2021)
- [Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models](https://mlsanity.com/paper/2403.18814/) (2024)
- [Function Vectors in Large Language Models](https://mlsanity.com/paper/2310.15213/) (2023)
- [Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learning](https://mlsanity.com/paper/2305.14160/) (2023)
- [mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understanding](https://mlsanity.com/paper/2307.02499/) (2023)
- [ImageBind-LLM: Multi-modality Instruction Tuning](https://mlsanity.com/paper/2309.03905/) (2023)
- [LLaVA-Phi: Efficient Multi-Modal Assistant with Small Language Model](https://mlsanity.com/paper/2401.02330/) (2024)
- [MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devices](https://mlsanity.com/paper/2312.16886/) (2023)
- [Smooth Grad-CAM++: An Enhanced Inference Level Visualization Technique for Deep Convolutional Neural Network Models](https://mlsanity.com/paper/1908.01224/) (2019)

## Built on by (most cited first, 30 of 38)

- [Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces](https://mlsanity.com/paper/2412.14171/) (2024)
- [Seeing Clearly by Layer Two: Enhancing Attention Heads to Alleviate Hallucination in LVLMs](https://mlsanity.com/paper/2411.09968/) (2024)
- [Backdoor Cleaning without External Guidance in MLLM Fine-tuning](https://mlsanity.com/paper/2505.16916/) (2025)
- [VFlowOpt: A Token Pruning Framework for LMMs with Visual Information Flow-Guided Optimization](https://mlsanity.com/paper/2508.05211/) (2025)
- [Linear Mechanisms for Spatiotemporal Reasoning in Vision Language Models](https://mlsanity.com/paper/2601.12626/) (2026)
- [Performance Gap in Entity Knowledge Extraction Across Modalities in Vision Language Models](https://mlsanity.com/paper/2412.14133/) (2024)
- [VISTA: Enhancing Vision-Text Alignment in MLLMs via Cross-Modal Mutual Information Maximization](https://mlsanity.com/paper/2505.10917/) (2025)
- [Position: Reasoning After Perception Means Reasoning Without Vision](https://mlsanity.com/paper/2507.16863/) (2025)
- [V-SEAM: Visual Semantic Editing and Attention Modulating for Causal Interpretability of Vision-Language Models](https://mlsanity.com/paper/2509.14837/) (2025)
- [Revealing Perception and Generation Dynamics in LVLMs: Mitigating Hallucinations via Validated Dominance Correction](https://mlsanity.com/paper/2512.18813/) (2025)
- [SurgXBench: Explainable Vision-Language Model Benchmark for Surgery](https://mlsanity.com/paper/2505.10764/) (2025)
- [Context Tokens are Anchors: Understanding the Repetition Curse in dMLLMs from an Information Flow Perspective](https://mlsanity.com/paper/2601.20520/) (2026)
- [AdaToken-3D: Dynamic Spatial Gating for Efficient 3D Large Multimodal-Models Reasoning](https://mlsanity.com/paper/2505.12782/) (2025)
- [EchoVLM: Dynamic Mixture-of-Experts Vision-Language Model for Universal Ultrasound Intelligence](https://mlsanity.com/paper/2509.14977/) (2025)
- [Seeing It or Not? Interpretable Vision-aware Latent Steering to Mitigate Object Hallucinations](https://mlsanity.com/paper/2505.17812/) (2025)
- [Visual Enhanced Depth Scaling for Multimodal Latent Reasoning](https://mlsanity.com/paper/2604.10500/) (2026)
- [Aligning What Vision-Language Models See and Perceive with Adaptive Information Flow](https://mlsanity.com/paper/2604.15809/) (2026)
- [Reading Images Like Texts: Sequential Image Understanding in Vision-Language Models](https://mlsanity.com/paper/2509.19191/) (2025)
- [Unleashing the Intrinsic Visual Representation Capability of Multimodal Large Language Models](https://mlsanity.com/paper/2512.06281/) (2025)
- [Where Not to Learn: Prior-Aligned Training with Subset-based Attribution Constraints](https://mlsanity.com/paper/2602.07008/) (2026)
- [SAGE: Sink-Aware Grounded Decoding for Multimodal Hallucination Mitigation](https://mlsanity.com/paper/2603.27898/) (2026)
- [A Single Layer to Explain Them All:Understanding Massive Activations in Large Language Models](https://mlsanity.com/paper/2605.08504/) (2026)
- [Enhancing Multimodal Large Language Models Complex Reason via Similarity Computation](https://mlsanity.com/paper/2412.09817/) (2024)
- [Interpreting Social Bias in LVLMs via Information Flow Analysis and Multi-Round Dialogue Evaluation](https://mlsanity.com/paper/2505.21106/) (2025)
- [ART: Attention Replacement Technique to Improve Factuality in LLMs](https://mlsanity.com/paper/2604.06393/) (2026)
- [Diffusion-CAM: Faithful Visual Explanations for dMLLMs](https://mlsanity.com/paper/2604.11005/) (2026)
- [Measuring Cross-Modal Synergy: A Benchmark for VLM Explainability](https://mlsanity.com/paper/2605.22168/) (2026)
- [When Attention Collapses: Stage-Aware Visual Token Pruning from Structure to Semantics](https://mlsanity.com/paper/2606.03569/) (2026)
- [Where to Place the Query? Unveiling and Mitigating Positional Bias in In-Context Learning for Diffusion LLMs via Decoding Dynamics](https://mlsanity.com/paper/2606.19349/) (2026)
- [Toward Modality Gap: Vision Prototype Learning for Weakly-supervised Semantic Segmentation with CLIP](https://mlsanity.com/paper/2412.19650/) (2024)

## Abstract (arXiv)

Large Vision Language Models (LVLMs) achieve great performance on visual-language reasoning tasks, however, the black-box nature of LVLMs hinders in-depth research on the reasoning mechanism. As all images need to be converted into image tokens to fit the input format of large language models (LLMs) along with natural language prompts, sequential visual representation is essential to the performance of LVLMs, and the information flow analysis approach can be an effective tool for determining interactions between these representations. In this paper, we propose integrating attention analysis with LLaVA-CAM, concretely, attention scores highlight relevant regions during forward propagation, while LLaVA-CAM captures gradient changes through backward propagation, revealing key image features. By exploring the information flow from the perspective of visual representation contribution, we observe that it tends to converge in shallow layers but diversify in deeper layers. To validate our analysis, we conduct comprehensive experiments with truncation strategies across various LVLMs for visual question answering and image captioning tasks, and experimental results not only verify our hypothesis but also reveal a consistent pattern of information flow convergence in the corresponding layers, and the information flow cliff layer will be different due to different contexts. The paper's source code can be accessed from urlhttps://github.com/zhangbaijin/From-Redundancy-to-Relevance