mlsanityData to 7 Oct 2026

SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference

arXiv 2410.04417 ↗First posted 6 Oct 2024cs.CVcited by 378

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

In vision-language models (VLMs), visual tokens usually bear a significant amount of computational overhead despite sparsity of information in them when compared to text tokens. To address this, most existing methods learn a network to prune redundant visual tokens using certain training data. Differently, we propose a text-guided training-free token optimization mechanism dubbed SparseVLM that eliminates the need of extra parameters or fine-tuning costs. Given that visual tokens complement text tokens in VLM's linguistic reasoning, we select relevant text tokens to rate the significance of visual tokens using self-attention matrices and, then, prune visual tokens using the proposed strategy to maximize sparsity while retaining information. In particular, we introduce a rank-based strategy to adaptively determine the sparsification ratio for each layer, alongside a token recycling method that compresses pruned tokens into more compact representations. Experimental results show that SparseVLM increases the efficiency of various VLMs in a number of image and video understanding tasks. For example, LLaVA when equipped with SparseVLM achieves 54% reduction in FLOPs, 37% decrease in CUDA latency while maintaining 97% of its original accuracy. Our code is available at https://github.com/Gumpest/SparseVLMs.

Key facts. Cited by 378 papers in the map (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 43 of its 58 references that are in the map. First posted on arXiv 6 Oct 2024.

Builds on

43 of 58 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024VideoLLM-MoD: Efficient Video-Language Streaming with Mixture-of-Depths Vision Computation29 Aug
  2. 2024VoCo-LLaMA: Towards Vision Compression with Large Language Models18 Jun
  3. 2024DeCo: Decoupling Token Compression from Semantic Abstraction in Multimodal Large Language Models31 May
  4. 2024Matryoshka Multimodal Models27 May
  5. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models22 Mar
  6. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models11 Mar

Other references · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  10. 2022A ConvNet for the 2020scited by 3,545
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
25 more
  1. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  2. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  3. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  4. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  5. 2019Towards VQA Models That Can Readcited by 1,696
  6. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  7. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  8. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  9. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  10. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  11. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  12. 2022Token Merging: Your ViT But Fastercited by 995
  13. 2023Instruction Tuning with GPT-4cited by 676
  14. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  15. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  16. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  17. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 479
  18. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  19. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 366
  20. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  21. 2023Honeybee: Locality-enhanced Projector for Multimodal LLMcited by 195
  22. 2023FreeKD: Knowledge Distillation via Semantic Frequency Promptcited by 27
  23. 2024Unveiling the Tapestry of Consistency in Large Vision-Language Modelscited by 22
  24. 2024TCFormer: Visual Recognition via Token Clustering Transformercited by 12
  25. 2024Decomposing the Neurons: Activation Sparsity via Mixture of Experts for Continual Test Time Adaptationcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 19; 2025: 157; 2026: 202.

378 papers.

Sort:

By area:

  1. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Modelscited by 298
  2. 2024PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reductioncited by 283
  3. 2024Beyond Text-Visual Attention: Exploiting Visual Cues for Effective Token Pruning in VLMscited by 153
  4. 2025Stop Looking for Important Tokens in Multimodal Language Models: Duplication Matters Morecited by 149
  5. 2025Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMscited by 92
  6. 2024ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Modelscited by 89
  7. 2025EfficientVLA: Training-Free Acceleration and Compression for Vision-Language-Action Modelscited by 89
  8. 2025PD-VLA: Accelerating Vision-Language-Action Model Integrated with Action Chunking via Parallel Decodingcited by 88
  9. 2025VLA-Cache: Efficient Vision-Language-Action Manipulation via Adaptive Token Cachingcited by 83
  10. 2024FastVLM: Efficient Vision Encoding for Vision Language Modelscited by 76
  11. 2025Token Pruning in Multimodal Large Language Models: Are We Solving the Right Problem?cited by 76
  12. 2025MoLe-VLA: Dynamic Layer-skipping Vision Language Action Model via Mixture-of-Layers for Efficient Robot Manipulationcited by 69
38 more
  1. 2024Multi-Stage Vision Token Dropping: Towards Efficient Multimodal Large Language Modelcited by 67
  2. 2024Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsificationcited by 59
  3. 2025SP-VLA: A Joint Model Scheduling and Token Pruning Approach for VLA Model Accelerationcited by 56
  4. 20254D-VLA: Spatiotemporal Vision-Language-Action Pretraining with Cross-Scene Calibrationcited by 55
  5. 2025StreamMem: Query-Agnostic KV Cache Memory for Streaming Video Understandingcited by 53
  6. 2024FrameFusion: Combining Similarity and Importance for Video Token Reduction on Large Vision Language Modelscited by 52
  7. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 49
  8. 2025FastVID: Dynamic Density Pruning for Fast Video Large Language Modelscited by 47
  9. 2025AdaReTaKe: Adaptive Redundancy Reduction to Perceive Longer for Video-language Understandingcited by 45
  10. 2025Don't Just Chase "Highlighted Tokens" in MLLMs: Revisiting Visual Holistic Context Retentioncited by 45
  11. 2024AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruningcited by 43
  12. 2025VScan: Rethinking Visual Token Reduction for Efficient Large Vision-Language Modelscited by 42
  13. 2025VisionThink: Smart and Efficient Vision Language Model via Reinforcement Learningcited by 41
  14. 2025SpecPrune-VLA: Accelerating Vision-Language-Action Models via Action-Aware Self-Speculative Pruningcited by 39
  15. 2024Filter, Correlate, Compress: Training-Free Token Reduction for MLLM Accelerationcited by 38
  16. 2025Video Compression Commander: Plug-and-Play Inference Acceleration for Video Large Language Modelscited by 38
  17. 2025CEED-VLA: Consistency Vision-Language-Action Model with Early-Exit Decodingcited by 38
  18. 2024A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMscited by 35
  19. 2025When Large Vision-Language Model Meets Large Remote Sensing Imagery: Coarse-to-Fine Text-Guided Token Pruningcited by 35
  20. 2025The Better You Learn, The Smarter You Prune: Towards Efficient Vision-language-action Models via Differentiable Token Pruningcited by 34
  21. 2024ReTaKe: Reducing Temporal and Knowledge Redundancy for Long Video Understandingcited by 33
  22. 2025Global Compression Commander: Plug-and-Play Inference Acceleration for High-Resolution Large Vision-Language Modelscited by 32
  23. 2025Similarity-Aware Token Pruning: Your VLM but Fastercited by 32
  24. 2025Action-aware Dynamic Pruning for Efficient Vision-Language-Action Manipulationcited by 32
  25. 2025FOLDER: Accelerating Multi-modal Large Language Models with Enhanced Performancecited by 31
  26. 2025Spec-VLA: Speculative Decoding for Vision-Language-Action Models with Relaxed Acceptancecited by 31
  27. 2025Think Twice, Act Once: Token-Aware Compression and Action Reuse for Efficient Inference in Vision-Language-Action Modelscited by 30
  28. 2025Shifting AI Efficiency From Model-Centric to Data-Centric Compressioncited by 29
  29. 2025FlowCut: Rethinking Redundancy via Information Flow for Efficient Vision-Language Modelscited by 29
  30. 2025MMTok: Multimodal Coverage Maximization for Efficient Inference of VLMscited by 29
  31. 2025Spatial Understanding from Videos: Structured Prompts Meet Simulation Datacited by 28
  32. 2025Scaling Vision Pre-Training to 4K Resolutioncited by 26
  33. 2025MoVE-KD: Knowledge Distillation for VLMs with Mixture of Visual Encoderscited by 25
  34. 2025LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMscited by 25
  35. 2025Variation-aware Vision Token Dropping for Faster Large Vision-Language Modelscited by 25
  36. 2025StreamKV: Streaming Video Question-Answering with Segment-based KV Cache Retrieval and Compressioncited by 24
  37. 2025SCOPE: Saliency-Coverage Oriented Token Pruning for Efficient Multimodel LLMscited by 23
  38. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 22

50 most cited of 378 All 378 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Inference Optimal VLMs Need Fewer Visual Tokens and More Parameters
  2. 2025TokenFLEX: Unified VLM Training for Flexible Visual Tokens Inference
  3. 2026SlimVLM: Sensitivity-aware Dynamic Structured Pruning with Adaptive Visual Token Selection for Efficient Vision-Language Models
  4. 2026PACE: A Unified Condense-and-Extract Paradigm for Fast VLM Inference
  5. 2024Efficient Vision-Language Models by Summarizing Visual Tokens into Compact Registers
  6. 2024ZipVL: Efficient Large Vision-Language Models with Dynamic Token Sparsification
  7. 2026SinkPruner: Sink-Free Visual Token Pruning for Multimodal Large Language Models
  8. 2026ResPrune: Text-Conditioned Subspace Reconstruction for Visual Token Pruning in Large Vision-Language Models