mlsanityData to 7 Oct 2026

VisiPruner: Decoding Discontinuous Cross-Modal Dynamics for Efficient Multimodal LLMs

arXiv 2510.17205 ↗First posted 20 Oct 2025cs.CV cs.CLcited by 14

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have achieved strong performance across vision-language tasks, but suffer from significant computational overhead due to the quadratic growth of attention computations with the number of multimodal tokens. Though efforts have been made to prune tokens in MLLMs, textitthey lack a fundamental understanding of how MLLMs process and fuse multimodal information. Through systematic analysis, we uncover a three-stage cross-modal interaction process: (1) Shallow layers recognize task intent, with visual tokens acting as passive attention sinks; (2) Cross-modal fusion occurs abruptly in middle layers, driven by a few critical visual tokens; (3) Deep layers discard vision tokens, focusing solely on linguistic refinement. Based on these findings, we propose VisiPruner, a training-free pruning framework that reduces up to 99% of vision-related attention computations and 53.9% of FLOPs on LLaVA-v1.5 7B. It significantly outperforms existing token pruning methods and generalizes across diverse MLLMs. Beyond pruning, our insights further provide actionable guidelines for training efficient MLLMs by aligning model architecture with its intrinsic layer-wise processing dynamics. Our code is available at: https://github.com/EIT-NLP/VisiPruner.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 43 of its 48 references that are in the map. First posted on arXiv 20 Oct 2025.

Builds on

43 of 48 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025Multimodal Language Models See Better When They Look Shallower30 Apr
  2. 2025Multi-Layer Visual Feature Fusion in Multimodal LLMs: Methods, Analysis, and Best Practices8 Mar
  3. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling6 Dec
  4. 2024To Preserve or To Compress: An In-Depth Study of Connector Selection in Multimodal Large Language Models9 Oct
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  6. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models11 Nov
  7. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  8. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  9. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun

Other references · 34

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  7. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  8. 2019Towards VQA Models That Can Readcited by 1,696
  9. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  10. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  11. 2023Vision Transformers Need Registerscited by 844
  12. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
22 more
  1. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 629
  2. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  3. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  4. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Modelscited by 394
  5. 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inferencecited by 378
  6. 2021AudioCLIP: Extending CLIP to Image, Text and Audiocited by 335
  7. 2024PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reductioncited by 283
  8. 2024MobileVLM V2: Faster and Stronger Baseline for Vision Language Modelcited by 172
  9. 2025LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Tokencited by 164
  10. 2022Analyzing Transformers in Embedding Spacecited by 131
  11. 2024Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language Modelscited by 125
  12. 2024Towards Interpreting Visual Information Processing in Vision-Language Modelscited by 115
  13. 2019GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answeringcited by 77
  14. 2024Cross-modal Information Flow in Multimodal Large Language Modelscited by 65
  15. 2024HiRED: Attention-Guided Token Dropping for Efficient Inference of High-Resolution Vision-Language Modelscited by 27
  16. 2024Treat Visual Tokens as Text? But Your MLLM Only Needs Fewer Efforts to Seecited by 21
  17. 2024Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findingscited by 15
  18. 2025SkipGPT: Dynamic Layer Pruning Reinvented with Token Awareness and Module Decouplingcited by 11
  19. 2025LLM as Effective Streaming Processor: Bridging Streaming-Batch Mismatches with Group Position Encodingcited by 9
  20. 2024Unveiling In-Context Learning: A Coordinate System to Understand Its Working Mechanismcited by 6
  21. 2025RedundancyLens: Revealing and Exploiting Visual Token Processing Redundancy for Efficient Decoder-Only MLLMscited by 6
  22. 2025Context Guided Transformer Entropy Modeling for Video Compressioncited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 12.

14 papers.

Sort:

By area:

  1. 2026HiDrop: Hierarchical Vision Token Reduction in MLLMs via Late Injection, Concave Pyramid Pruning, and Early Exitcited by 15
  2. 2026ViCA: Efficient Multimodal LLMs with Vision-Only Cross-Attentioncited by 8
  3. 2026Think-as-You-See: Streaming Chain-of-Thought Reasoning for Large Vision-Language Modelscited by 6
  4. 2025The Few Govern the Many:Unveiling Few-Layer Dominance for Time Series Modelscited by 3
  5. 2026MI-Pruner: Crossmodal Mutual Information-guided Token Pruner for Efficient MLLMscited by 2
  6. 2026Accelerating Multimodal Large Language Models with Prior-Corrected Token Reductioncited by 2
  7. 2026EvoCut: Multi-Layer Evolution-Aware Visual Token Compression for Efficient Large Vision-Language Modelscited by 1
  8. 2026Spectral Evolution-Guided Token Pruning in Multimodal Large Language Modelscited by 1
  9. 2025ShaRP: SHAllow-LayeR Pruning for Efficient Video Large Language Modelscited by 0
  10. 2026Ego: Embedding-Guided Personalization of Vision-Language Modelscited by 0
  11. 2026YARD: Y-Architecture Register Decoding for Efficient Hallucination Mitigation in Large Vision-Language Modelscited by 0
  12. 2026ProactiveLLM: Learning Active Interaction for Streaming Large Language Modelscited by 0
2 more
  1. 2026MLLMs Get It Right, Then Get It Wrong: Tracing and Correcting Late-Layer Textual Biascited by 0
  2. 2026TruthLens: Object Hallucination Detection via Self-Evaluating Truthfulness Scores in LVLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMs
  2. 2025Lifting the Veil on Visual Information Flow in MLLMs: Unlocking Pathways to Faster Inference
  3. 2026Architecture-Dependent Fusion Pathways in MLLMs
  4. 2024EE-MLLM: A Data-Efficient and Compute-Efficient Multimodal Large Language Model
  5. 2026Probing Cross-modal Information Hubs in Audio-Visual LLMs
  6. 2026Where Does Vision Meet Language? Understanding and Refining Visual Fusion in MLLMs via Contrastive Attention
  7. 2026Measuring Cross-Modal Synergy: A Benchmark for VLM Explainability
  8. 2025Rethinking Visual Token Reduction in LVLMs Under Cross-Modal Misalignment