mlsanityData to 14 Oct 2026

PACT: Pruning and Clustering-Based Token Reduction for Faster Visual Language Models

arXiv 2504.08966 ↗First posted 11 Apr 2025cs.CVcited by 49

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

also in 1 more
Abstract (arXiv)

Visual Language Models require substantial computational resources for inference due to the additional input tokens needed to represent visual information. However, these visual tokens often contain redundant and unimportant information, resulting in an unnecessarily high number of tokens. To address this, we introduce PACT, a method that reduces inference time and memory usage by pruning irrelevant tokens and merging visually redundant ones at an early layer of the language model. Our approach uses a novel importance metric to identify unimportant tokens without relying on attention scores, making it compatible with FlashAttention. We also propose a novel clustering algorithm, called Distance Bounded Density Peak Clustering, which efficiently clusters visual tokens while constraining the distances between elements within a cluster by a predefined threshold. We demonstrate the effectiveness of PACT through extensive experiments.

Key facts. Cited by 49 papers in the map (data through 14 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 47 of its 59 references that are in the map. First posted on arXiv 11 Apr 2025.

Builds on

47 of 59 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024HiRED: Attention-Guided Token Dropping for Efficient Inference of High-Resolution Vision-Language Models20 Aug
  2. 2024Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference9 May
  3. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models22 Mar
  4. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models11 Mar

Other references · 43

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  7. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  8. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  9. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  10. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  11. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  12. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
31 more
  1. 2019Towards VQA Models That Can Readcited by 1,718
  2. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  3. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  4. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  5. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  6. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  7. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  8. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  9. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  10. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  11. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  12. 2022Token Merging: Your ViT But Fastercited by 1,021
  13. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  14. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  15. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  16. 2016A Diagram Is Worth A Dozen Imagescited by 804
  17. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 680
  18. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
  19. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 586
  20. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 561
  21. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  22. 2021InfographicVQAcited by 437
  23. 2021OCR-free Document Understanding Transformercited by 407
  24. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 363
  25. 2023Perception Test: A Diagnostic Benchmark for Multimodal Video Modelscited by 314
  26. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
  27. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  28. 2024MuirBench: A Comprehensive Benchmark for Robust Multi-image Understandingcited by 188
  29. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 187
  30. 2024Vibe-Eval: A hard evaluation suite for measuring progress of multimodal language modelscited by 35
  31. 2023DocParser: End-to-end OCR-free Information Extraction from Visually Rich Documentscited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 15; 2026: 34.

49 papers.

Sort:

By area:

  1. 2025LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMscited by 26
  2. 2025FastDriveVLA: Efficient End-to-End Driving via Plug-and-Play Reconstruction-based Token Pruningcited by 20
  3. 2025A Glimpse to Compress: Dynamic Visual Token Pruning for Large Vision-Language Modelscited by 18
  4. 2025Prune2Drive: A Plug-and-Play Framework for Accelerating Vision-Language Models in Autonomous Drivingcited by 16
  5. 2026IVC-Prune: Revealing the Implicit Visual Coordinates in LVLMs for Vision Token Pruningcited by 12
  6. 2026ViCA: Efficient Multimodal LLMs with Vision-Only Cross-Attentioncited by 10
  7. 2026Language-Guided Token Compression with Reinforcement Learning in Large Vision-Language Modelscited by 10
  8. 2026ResidualKV: Residual-Based KV Cache Compression for Efficient Long-Context Inferencecited by 9
  9. 2026Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospectscited by 9
  10. 2025SalQ-VLM: Fine-Grained Saliency-Guided Quantization for Vision-Language Modelscited by 8
  11. 2025LoC-Path: Learning to Compress for Pathology Multimodal Large Language Modelscited by 6
  12. 2026Photon: Speedup Volume Understanding with Efficient Multimodal Large Language Modelscited by 6
37 more
  1. 2026IDPruner: Harmonizing Importance and Diversity in Visual Token Pruning for MLLMscited by 5
  2. 2026Seeing the End at Step Zero: Accelerating Diffusion MLLMs via MLP Sparsity-Aware Truncationcited by 5
  3. 2025PPE: Positional Preservation Embedding for Token Compression in Multimodal Large Language Modelscited by 4
  4. 2026ForestPrune: High-ratio Visual Token Compression for Video Multimodal Large Language Models via Spatial-Temporal Forest Modelingcited by 4
  5. 2025TransPrune: Token Transition Pruning for Efficient Large Vision-Language Modelcited by 3
  6. 2026Magic-MM-Embedding: Towards Visual-Token-Efficient Universal Multimodal Embedding with MLLMscited by 3
  7. 2026OccamToken: Efficient VLM Inference with Training-Free and Budget-Adaptive Token Pruningcited by 3
  8. 2025DeepInsert: Early Layer Bypass for Efficient and Performant Multimodal Understandingcited by 2
  9. 2025STAR: Stage-Wise Attention-Guided Token Reduction for Efficient Large Vision-Language Models Inferencecited by 2
  10. 2026RCP: Representation Consistency Pruner for Mitigating Distribution Shift in Large Vision-Language Modelscited by 2
  11. 2025ReGATE: Learning Faster and Better with Fewer Tokens in MLLMscited by 1
  12. 2025GM-Skip: Metric-Guided Transformer Block Skipping for Efficient Vision-Language Modelscited by 1
  13. 2025Focus Through Motion: RGB-Event Collaborative Token Sparsification for Efficient Object Detectioncited by 1
  14. 2025Towards Lossless Ultimate Vision Token Compression for VLMscited by 1
  15. 2026ST-Prune: Training-Free Spatio-Temporal Token Pruning for Vision-Language Models in Autonomous Drivingcited by 1
  16. 2026OTT-Vid: Optimal Transport Temporal Token Compression for Video Large Language Modelscited by 1
  17. 2026AQuaUI: Visual Token Reduction for GUI Agents with Adaptive Quadtreescited by 1
  18. 2026Baseline-Relative Counterfactual Refinement for Bit-Aware Visual Token Communicationcited by 1
  19. 2026S2Prune: Spatially Structured Visual Token Pruning for Multimodal Large Language Modelscited by 1
  20. 2025EventSTU: Event-Guided Efficient Spatio-Temporal Understanding for Video Large Language Modelscited by 0
  21. 2026ConsensusDrop: Fusing Visual and Cross-Modal Saliency for Efficient Vision Language Modelscited by 0
  22. 2026Long-range Modeling and Processing of Multimodal Event Sequencescited by 0
  23. 2026Seeing Beyond Redundancy: Task Complexity's Role in Vision Token Specialization in VLLMscited by 0
  24. 2026IWP: Token Pruning as Implicit Weight Pruning in Large Vision Language Modelscited by 0
  25. 2026Sink-Token-Aware Pruning for Fine-Grained Video Understanding in Efficient Video LLMscited by 0
  26. 2026VisMMOE: Exploiting Visual-Expert Affinity for Efficient Visual-Language MoE Offloadingcited by 0
  27. 2026Focus-then-Context: Subject-Centric Progressive Visual Token Reduction for Vision-Language Modelscited by 0
  28. 2026ETC: Extreme Token Compression via Task-aware Visual Information Distillation in VLMscited by 0
  29. 2026MVPruner: Dynamic Token Pruning for Accelerating Multi-view Vision-Language Models in Autonomous Drivingcited by 0
  30. 2026GSTEP: Global Spatio-Temporal Density-Driven Visual Token Pruning for Efficient Video Large Language Modelscited by 0
  31. 2026CARVE: Cross-Slice Anisotropic Reallocation of Visual Evidence for Efficient 3D Medical Volume Understandingcited by 0
  32. 2026ZIPBrain: Can EEG Foundation Models Be Faster, Locally Deployable, but Accurate?cited by 0
  33. 2026Aggregating Visual Information with Optimal Transport for VideoLM Token Compressioncited by 0
  34. 2026SLICEChat: Progressive In-Encoder Token Pruning for Whole-Slide Pathology Language Modelscited by 0
  35. 2026VisionPsy-Nano: Improving Accuracy, Efficiency, and Reliability in On-Device Vision-Language Modelscited by 0
  36. 2026Still There, No Longer Seen: Exposing Compression-Induced Risk in Large Vision-Language Modelscited by 0
  37. 2026When Masking Helps or Hurts Robustness in Compressed CLIP: A Pre-Deployment Diagnosticcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026SinkPruner: Sink-Free Visual Token Pruning for Multimodal Large Language Models
  2. 2026How Many Visual Tokens Do Multimodal Language Models Need? Scaling Visual Token Pruning with F^3A
  3. 2025ToDRE: Effective Visual Token Pruning via Token Diversity and Task Relevance
  4. 2025When Token Pruning is Worse than Random: Understanding Visual Token Information in VLLMs
  5. 2026EvoPrune: Early-Stage Visual Token Pruning for Efficient MLLMs
  6. 2024Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language Models
  7. 2026The Model Knows Which Tokens Matter: Automatic Token Selection via Noise Gating
  8. 2025Beyond Intermediate States: Explaining Visual Redundancy through Language