mlsanityData to 7 Oct 2026

Beyond Text-Visual Attention: Exploiting Visual Cues for Effective Token Pruning in VLMs

arXiv 2412.01818 ↗First posted 2 Dec 2024cs.CV cs.AIcited by 153

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

also in 1 more
Abstract (arXiv)

Large vision-language models (LVLMs) generally contain significantly more visual tokens than their textual counterparts, resulting in a considerable computational burden. Recent efforts have been made to tackle this issue by pruning visual tokens early within the language model. Most existing works use attention scores between text and visual tokens to assess the importance of visual tokens. However, in this study, we first analyze the text-visual attention in the language model and find that this score is not an ideal indicator for token pruning. Based on the analysis, We propose VisPruner, a plug-and-play method that utilizes visual cues for more effective token pruning in LVLMs. Specifically, we first use visual attention to select a limited number of significant tokens. Then, we remove duplicate tokens from the remaining ones based on their similarity. By retaining diverse tokens alongside the initially selected important tokens, we maximally preserve the visual information of the input image. Experimental results demonstrate that our VisPruner sustains strong performance across various VLM architectures and reduction ratios, significantly outperforming existing methods based on text-visual attention. Notably, without any training, VisPruner can reduce the FLOPs of LLaVA-1.5-7B by 91% and inference latency by 75%, while maintaining comparable performance. Our code is available at https://github.com/Theia-4869/VisPruner.

Key facts. Cited by 153 papers in the map (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 55 of its 73 references that are in the map. First posted on arXiv 2 Dec 2024.

Builds on

55 of 73 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reduction22 Oct
  2. 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference6 Oct
  3. 2024Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language Models16 Sep
  4. 2024TokenPacker: Efficient Visual Projector for Multimodal LLM2 Jul
  5. 2024Efficient Large Multi-modal Models via Visual Context Compression28 Jun
  6. 2024Matryoshka Query Transformer for Large Vision-Language Models29 May
  7. 2024Matryoshka Multimodal Models27 May
  8. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models22 Mar
  9. 2024LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Images18 Mar
  10. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models11 Mar

Other references · 45

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2022Training language models to follow instructions with human feedbackcited by 15,712
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  12. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
33 more
  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  3. 2023Qwen Technical Reportcited by 3,273
  4. 2023Mistral 7Bcited by 3,065
  5. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  6. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  7. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  9. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  10. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  11. 2019Towards VQA Models That Can Readcited by 1,696
  12. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  13. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  14. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  15. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  16. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  17. 2022Token Merging: Your ViT But Fastercited by 995
  18. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  19. 2023Vision Transformers Need Registerscited by 844
  20. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  21. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  22. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  23. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  24. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  25. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 468
  26. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  27. 2020Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processingcited by 161
  28. 2016TGIF: A New Dataset and Benchmark on Animated GIF Descriptioncited by 124
  29. 2024Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language Modelscited by 112
  30. 2022Efficient Transformers with Dynamic Token Poolingcited by 75
  31. 2024MC-LLaVA: Multi-Concept Personalized Vision-Language Modelcited by 53
  32. 2022Pyramid-BERT: Reducing Complexity via Successive Core-set based Token Selectioncited by 13
  33. 2024On the token distance modeling ability of higher RoPE attention dimensioncited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 38; 2026: 114.

153 papers.

Sort:

By area:

  1. 2025Latent Visual Reasoningcited by 99
  2. 2025Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMscited by 92
  3. 2025StreamMem: Query-Agnostic KV Cache Memory for Streaming Video Understandingcited by 53
  4. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 49
  5. 2025The Better You Learn, The Smarter You Prune: Towards Efficient Vision-language-action Models via Differentiable Token Pruningcited by 34
  6. 2025Global Compression Commander: Plug-and-Play Inference Acceleration for High-Resolution Large Vision-Language Modelscited by 32
  7. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 22
  8. 2025InternVL-X: Advancing and Accelerating InternVL Series with Efficient Visual Token Compressioncited by 21
  9. 2025FastDriveVLA: Efficient End-to-End Driving via Plug-and-Play Reconstruction-based Token Pruningcited by 20
  10. 2025DocPruner: A Storage-Efficient Framework for Multi-Vector Visual Document Retrieval via Adaptive Patch-Level Embedding Pruningcited by 19
  11. 2025A Glimpse to Compress: Dynamic Visual Token Pruning for Large Vision-Language Modelscited by 18
  12. 2025Growing a Multi-head Twig via Distillation and Reinforcement Learning to Accelerate Large Vision-Language Modelscited by 15
38 more
  1. 2025SQAP-VLA: A Synergistic Quantization-Aware Pruning Framework for High-Performance Vision-Language-Action Modelscited by 15
  2. 2026FlashVID: Efficient Video Large Language Models via Training-free Tree-based Spatiotemporal Token Mergingcited by 14
  3. 2025FlashVLM: Text-Guided Visual Token Selection for Large Multimodal Modelscited by 11
  4. 2026IVC-Prune: Revealing the Implicit Visual Coordinates in LVLMs for Vision Token Pruningcited by 11
  5. 2025VideoScan: Enabling Efficient Streaming Video Understanding via Frame-level Semantic Carrierscited by 10
  6. 2025HiPrune: Hierarchical Attention for Efficient Token Pruning in Vision-Language Modelscited by 10
  7. 2026Language-Guided Token Compression with Reinforcement Learning in Large Vision-Language Modelscited by 9
  8. 2026ViCA: Efficient Multimodal LLMs with Vision-Only Cross-Attentioncited by 8
  9. 2026Sculpting the Vector Space: Towards Efficient Multi-Vector Visual Document Retrieval via Prune-then-Merge Frameworkcited by 8
  10. 2026Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospectscited by 8
  11. 2025Beyond Intermediate States: Explaining Visual Redundancy through Languagecited by 7
  12. 2025LLMC+: Benchmarking Vision-Language Model Compression with a Plug-and-play Toolkitcited by 7
  13. 2025MergeMix: A Unified Augmentation Paradigm for Visual and Multi-Modal Understandingcited by 7
  14. 2025Compressor-VLA: Instruction-Guided Visual Token Compression for Efficient Robotic Manipulationcited by 7
  15. 2026AgilePruner: An Empirical Study of Attention and Diversity for Adaptive Visual Token Pruning in Large Vision-Language Modelscited by 7
  16. 2026LearnPruner: Rethinking Attention-based Token Pruning in Vision Language Modelscited by 7
  17. 2026Unified Spatiotemporal Token Compression for Video-LLMs at Ultra-Low Retentioncited by 6
  18. 2026Photon: Speedup Volume Understanding with Efficient Multimodal Large Language Modelscited by 6
  19. 2026On the Adversarial Robustness of Large Vision-Language Models under Visual Token Compressioncited by 5
  20. 2026AngelSlim: A more accessible, comprehensive, and efficient toolkit for large model compressioncited by 5
  21. 2026EvoPrune: Early-Stage Visual Token Pruning for Efficient MLLMscited by 5
  22. 2026Large Vision-Language Models Get Lost in Attentioncited by 5
  23. 2026Improving Visual Token Reduction via Rectifying Distortions for Efficient Multimodal LLM Inferencecited by 5
  24. 2025AdaFV: Rethinking of Visual-Language alignment for VLM accelerationcited by 4
  25. 2025AirCache: Activating Inter-modal Relevancy KV Cache Compression for Efficient Large Vision-Language Model Inferencecited by 4
  26. 2025HiViS: Hiding Visual Tokens from the Drafter for Speculative Decoding in Vision-Language Modelscited by 4
  27. 2025EM-KD: Distilling Efficient Multimodal Large Language Model with Unbalanced Vision Tokenscited by 4
  28. 2025EchoingPixels: Aliasing-Resistant Joint Token Reduction for Audio-Visual LLMscited by 4
  29. 2026IDPruner: Harmonizing Importance and Diversity in Visual Token Pruning for MLLMscited by 4
  30. 2026Energy-Driven Adaptive Visual Token Pruning for Efficient Vision-Language Modelscited by 4
  31. 2026AVIS: Adaptive Test-Time Scaling for Vision-Language Modelscited by 4
  32. 2026Seeing the End at Step Zero: Accelerating Diffusion MLLMs via MLP Sparsity-Aware Truncationcited by 4
  33. 2025TransPrune: Token Transition Pruning for Efficient Large Vision-Language Modelcited by 3
  34. 2025Revisiting MLLM Token Technology through the Lens of Classical Visual Codingcited by 3
  35. 2025Attention Debiasing for Token Pruning in Vision Language Modelscited by 3
  36. 2025HIVTP: A Training-Free Method to Improve VLMs Efficiency via Hierarchical Visual Token Pruning Using Middle-Layer-Based Importance Scorecited by 3
  37. 2026Magic-MM-Embedding: Towards Visual-Token-Efficient Universal Multimodal Embedding with MLLMscited by 3
  38. 2026Focus-Scan-Refine: From Human Visual Perception to Efficient Visual Token Pruningcited by 3

50 most cited of 153 All 153 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Don't Just Chase "Highlighted Tokens" in MLLMs: Revisiting Visual Holistic Context Retention
  2. 2026Multi-Image Visual Token Pruning in Large Visual Language Models
  3. 2025Multi-Cue Adaptive Visual Token Pruning for Large Vision-Language Models
  4. 2026SlimVLM: Sensitivity-aware Dynamic Structured Pruning with Adaptive Visual Token Selection for Efficient Vision-Language Models
  5. 2025ToDRE: Effective Visual Token Pruning via Token Diversity and Task Relevance
  6. 2026TReVS: Integrating Textual Relevance and Visual Saliency for Efficient Vision-Language Model Token Pruning
  7. 2025VScan: Rethinking Visual Token Reduction for Efficient Large Vision-Language Models
  8. 2025STAR: Stage-Wise Attention-Guided Token Reduction for Efficient Large Vision-Language Models Inference