mlsanityData to 7 Oct 2026

Similarity-Aware Token Pruning: Your VLM but Faster

arXiv 2503.11549 ↗First posted 14 Mar 2025cs.CVcited by 32

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

also in 1 more
Abstract (arXiv)

The computational demands of Vision Transformers (ViTs) and Vision-Language Models (VLMs) remain a significant challenge due to the quadratic complexity of self-attention. While token pruning offers a promising solution, existing methods often introduce training overhead or fail to adapt dynamically across layers. We present SAINT, a training-free token pruning framework that leverages token similarity and a graph-based formulation to dynamically optimize pruning rates and redundancy thresholds. Through systematic analysis, we identify a universal three-stage token evolution process (aligner-explorer-aggregator) in transformers, enabling aggressive pruning in early stages without sacrificing critical information. For ViTs, SAINT doubles the throughput of ViT-H/14 at 224px with only 0.6% accuracy loss on ImageNet-1K, surpassing the closest competitor by 0.8%. For VLMs, we apply SAINT in three modes: ViT-only, LLM-only, and hybrid. SAINT reduces LLaVA-13B's tokens by 75%, achieving latency comparable to LLaVA-7B with less than 1% performance loss across benchmarks. Our work establishes a unified, practical framework for efficient inference in ViTs and VLMs.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 40 of its 43 references that are in the map. First posted on arXiv 14 Mar 2025.

Builds on

40 of 43 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Models5 Dec
  2. 2024PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reduction22 Oct
  3. 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference6 Oct
  4. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models11 Mar

Other references · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  7. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  10. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  12. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
24 more
  1. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  2. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  3. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  4. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  5. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  6. 2022Token Merging: Your ViT But Fastercited by 995
  7. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  8. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  9. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  10. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  11. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  12. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
  13. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 359
  14. 2024What matters when building vision-language models?cited by 321
  15. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  16. 2021Adaptive Token Sampling For Efficient Vision Transformerscited by 181
  17. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruningcited by 165
  18. 2023Token Fusion: Bridging the Gap between Token Pruning and Token Mergingcited by 123
  19. 2023What Do Self-Supervised Vision Transformers Learn?cited by 90
  20. 2023Joint Token Pruning and Squeezing Towards More Aggressive Compression of Vision Transformerscited by 79
  21. 2023OtterHD: A High-Resolution Multi-modality Modelcited by 72
  22. 2023Zero-TPrune: Zero-Shot Token Pruning through Leveraging of the Attention Graph in Pre-Trained Transformerscited by 61
  23. 2023PPT: Token Pruning and Pooling for Efficient Vision Transformerscited by 37
  24. 2023GTP-ViT: Efficient Vision Transformers via Graph-based Token Propagationcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 22.

32 papers.

Sort:

By area:

  1. 2025Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMscited by 92
  2. 2025Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmarkcited by 37
  3. 2025Object Detection with Multimodal Large Vision-Language Models: An In-depth Reviewcited by 14
  4. 2026N"uwa: Mending the Spatial Integrity Torn by VLM Token Pruningcited by 13
  5. 2026Beyond Surrogate Gradients: Fully Differentiable Token Pruning for Vision-Language Modelscited by 7
  6. 2025D3ToM: Decider-Guided Dynamic Token Merging for Accelerating Diffusion MLLMscited by 5
  7. 2025VLM-Pruner: Buffering for Spatial Sparsity in an Efficient VLM Centrifugal Token Pruning Paradigmcited by 5
  8. 2026AVIS: Adaptive Test-Time Scaling for Vision-Language Modelscited by 4
  9. 2026HAWK: Head Importance-Aware Visual Token Pruning in Multimodal Modelscited by 3
  10. 2026RetentiveKV: State-Space Memory for Uncertainty-Aware Multimodal KV Cache Evictioncited by 3
  11. 2025Reading Images Like Texts: Sequential Image Understanding in Vision-Language Modelscited by 2
  12. 2025Script: Graph-Structured and Query-Conditioned Semantic Token Pruning for Multimodal Large Language Modelscited by 2
20 more
  1. 2026Look Where It Matters: High-Resolution Crops Retrieval for Efficient VLMscited by 2
  2. 2026MI-Pruner: Crossmodal Mutual Information-guided Token Pruner for Efficient MLLMscited by 2
  3. 2026DynaTok: Temporally Adaptive and Positional Bias-Aware Token Compression for Video-LLMscited by 2
  4. 2026Accelerating Multimodal Large Language Models with Prior-Corrected Token Reductioncited by 2
  5. 2026Who Speaks for the Pruned? Visual Token Pruning as Coverage Optimizationcited by 2
  6. 2025Skip-It? Theoretical Conditions for Layer Skipping in Vision-Language Modelscited by 1
  7. 2025Seeing the Forest and the Trees: Query-Aware Tokenizer for Long-Video Multimodal Language Modelscited by 1
  8. 2026Spectral Evolution-Guided Token Pruning in Multimodal Large Language Modelscited by 1
  9. 2026DiffPrune: differentiable information throttling for token pruning in vision-language modelscited by 1
  10. 2026QCPruner: Query-Conditioned Population Coverage for Visual Token Pruningcited by 1
  11. 2025Parallel Vision Token Scheduling for Fast and Accurate Multimodal LMMs Inferencecited by 0
  12. 2026Taking Shortcuts for Categorical VQA Using Super Neuronscited by 0
  13. 2026VisMMOE: Exploiting Visual-Expert Affinity for Efficient Visual-Language MoE Offloadingcited by 0
  14. 2026RAPID: Layer-Wise Redundancy-Aware Pruning and Importance-Driven Token Merging for Efficient ViTcited by 0
  15. 2026DCP-Prune: Ultra-Low Token Pruning with Distribution Consistency Preservationcited by 0
  16. 2026Attention-Free and Lightweight Token Reduction for Efficient Vision-Language Modelscited by 0
  17. 2026From Token Importance to Conditional Removability: Rethinking Visual Token Pruning in Multimodal Large Language Modelscited by 0
  18. 2026SCOPD: Sparse-Context On-Policy Self-Distillation for Efficient Vision-Language Modelscited by 0
  19. 2026Text-Vision Synergistic Token Caching: A Training-Free Framework for Efficient Vision-Language-Action Inferencecited by 0
  20. 2026VisionWeave: Weaving Elastic Visual Representations as a Native Capability of MLLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Exploring Token Pruning in Vision State Space Models
  2. 2025Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimization
  3. 2024MADTP: Multimodal Alignment-Guided Dynamic Token Pruning for Accelerating Vision-Language Transformer
  4. 2022Beyond Attentive Tokens: Incorporating Token Importance and Diversity for Efficient Vision Transformers
  5. 2024VLTP: Vision-Language Guided Token Pruning for Task-Oriented Segmentation
  6. 2026Multi-Image Visual Token Pruning in Large Visual Language Models
  7. 2026Beyond One-Size-Fits-All: Sample-Adaptive Strategy Routing for Vision Token Pruning in MLLMs
  8. 2026Collaborative Multi-Mode Pruning for Vision-Language Models