mlsanityData to 7 Oct 2026

Lifting the Veil on Visual Information Flow in MLLMs: Unlocking Pathways to Faster Inference

arXiv 2503.13108 ↗First posted 17 Mar 2025cs.CV cs.AIcited by 23

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

Multimodal large language models (MLLMs) improve performance on vision-language tasks by integrating visual features from pre-trained vision encoders into large language models (LLMs). However, how MLLMs process and utilize visual information remains unclear. In this paper, a shift in the dominant flow of visual information is uncovered: (1) in shallow layers, strong interactions are observed between image tokens and instruction tokens, where most visual information is injected into instruction tokens to form cross-modal semantic representations; (2) in deeper layers, image tokens primarily interact with each other, aggregating the remaining visual information to optimize semantic representations within visual modality. Based on these insights, we propose Hierarchical Modality-Aware Pruning (HiMAP), a plug-and-play inference acceleration method that dynamically prunes image tokens at specific layers, reducing computational costs by approximately 65% without sacrificing performance. Our findings offer a new understanding of visual information processing in MLLMs and provide a state-of-the-art solution for efficient inference.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 41 of its 47 references that are in the map. First posted on arXiv 17 Mar 2025.

Builds on

41 of 47 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference9 May
  2. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models22 Mar
  3. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models11 Mar

Other references · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  11. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  12. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
26 more
  1. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  2. 2023A Survey of Large Language Modelscited by 2,537
  3. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  4. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  5. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  7. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  8. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  9. 2019Towards VQA Models That Can Readcited by 1,696
  10. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  11. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  12. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  13. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  14. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  15. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  16. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  17. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  18. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  19. 2019Are Sixteen Heads Really Better than One?cited by 803
  20. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  21. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  22. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  23. 2018nocaps: novel object captioning at scalecited by 444
  24. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 235
  25. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 219
  26. 2024Visual Anchors Are Strong Information Aggregators For Multimodal Large Language Modelcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 16.

23 papers.

Sort:

By area:

  1. 2025Visual Thoughts: A Unified Perspective of Understanding Multimodal Chain-of-Thoughtcited by 34
  2. 2026N"uwa: Mending the Spatial Integrity Torn by VLM Token Pruningcited by 13
  3. 2026ViCA: Efficient Multimodal LLMs with Vision-Only Cross-Attentioncited by 8
  4. 2025Tell Model Where to Look: Mitigating Hallucinations in MLLMs by Vision-Guided Attentioncited by 6
  5. 2025METEOR: Multi-Encoder Collaborative Token Pruning for Efficient Vision Language Modelscited by 3
  6. 2025Attention Debiasing for Token Pruning in Vision Language Modelscited by 3
  7. 2026Sparrow: Text-Anchored Window Attention with Visual-Semantic Glimpsing for Speculative Decoding in Video LLMscited by 3
  8. 2025Video Parallel Scaling: Aggregating Diverse Frame Subsets for VideoLLMscited by 2
  9. 2025Cross-Layer Vision Smoothing: Enhancing Visual Understanding via Sustained Focus on Key Objects in Large Vision-Language Modelscited by 2
  10. 2026MI-Pruner: Crossmodal Mutual Information-guided Token Pruner for Efficient MLLMscited by 2
  11. 2026Aligning What Vision-Language Models See and Perceive with Adaptive Information Flowcited by 2
  12. 2026Correcting Visual Blur Induced by Attention Distraction to Reduce Hallucinations: Algorithm and Theorycited by 2
11 more
  1. 2026Reroute, Don't Remove: Recoverable Visual Token Routing for Vision-Language Modelscited by 2
  2. 2026Do We Really Need External Tools to Mitigate Hallucinations? SIRA: Shared-Prefix Internal Reconstruction of Attributioncited by 1
  3. 2026One Layer's Trash is Another Layer's Treasure: Adaptive Layer-wise Visual Token Selection in LVLMscited by 1
  4. 2025ShaRP: SHAllow-LayeR Pruning for Efficient Video Large Language Modelscited by 0
  5. 2026Seeing Beyond Redundancy: Task Complexity's Role in Vision Token Specialization in VLLMscited by 0
  6. 2026Look Twice: Training-Free Evidence Highlighting for Knowledge-based Visual Question Answeringcited by 0
  7. 2026MLLMs Get It Right, Then Get It Wrong: Tracing and Correcting Late-Layer Textual Biascited by 0
  8. 2026Attending to Multimodal Generation One Token at a Timecited by 0
  9. 2026SepPrune:A Separator-based Pruning Framework for Efficient Multimodal Large Language Modelscited by 0
  10. 2026PromptPath: Prompt-Adaptive Computational Pathways for In-Context Learningcited by 0
  11. 2026Summarize Before Grounding: Query-Guided Chunk Condensation for Long-Video Temporal Groundingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospects
  2. 2026Vision Inference Former: Sustaining Visual Consistency in Multimodal Large Language Models
  3. 2024A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMs
  4. 2026When Vision Becomes Text: Visual Token Pruning via Cross-Modal Residual Guidance in VLMs
  5. 2026Late-Layer Fusion is Enough: Dual-Path Vision Token Routing for Multimodal Large Language Models under Visual Saturation
  6. 2026EvoPrune: Early-Stage Visual Token Pruning for Efficient MLLMs
  7. 2024ST3: Accelerating Multimodal Large Language Model by Spatial-Temporal Visual Token Trimming
  8. 2025Rethinking Visual Token Reduction in LVLMs Under Cross-Modal Misalignment