mlsanityData to 7 Oct 2026

DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Models

arXiv 2503.02175 ↗First posted 4 Mar 2025cs.CV cs.AI cs.LGcited by 177

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

also in 1 more
Abstract (arXiv)

Large Multimodal Models (LMMs) have emerged as powerful models capable of understanding various data modalities, including text, images, and videos. LMMs encode both text and visual data into tokens that are then combined and processed by an integrated Large Language Model (LLM). Including visual tokens substantially increases the total token count, often by thousands. The increased input length for LLM significantly raises the complexity of inference, resulting in high latency in LMMs. To address this issue, token pruning methods, which remove part of the visual tokens, are proposed. The existing token pruning methods either require extensive calibration and fine-tuning or rely on suboptimal importance metrics which results in increased redundancy among the retained tokens. In this paper, we first formulate token pruning as Max-Min Diversity Problem (MMDP) where the goal is to select a subset such that the diversity among the selected tokens is maximized. Then, we solve the MMDP to obtain the selected subset and prune the rest. The proposed method, DivPrune, reduces redundancy and achieves the highest diversity of the selected tokens. By ensuring high diversity, the selected tokens better represent the original tokens, enabling effective performance even at high pruning ratios without requiring fine-tuning. Extensive experiments with various LMMs show that DivPrune achieves state-of-the-art accuracy over 16 image- and video-language datasets. Additionally, DivPrune reduces both the end-to-end latency and GPU memory usage for the tested models. The code is available hrefhttps://github.com/vbdi/divprunehere.

Key facts. Cited by 177 papers in the map (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 44 of its 54 references that are in the map. First posted on arXiv 4 Mar 2025.

Builds on

44 of 54 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language Models16 Sep
  2. 2024TokenPacker: Efficient Visual Projector for Multimodal LLM2 Jul
  3. 2024Matryoshka Multimodal Models27 May
  4. 2024Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference9 May
  5. 2024On Speculative Decoding for Multimodal Large Language Models13 Apr
  6. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models22 Mar
  7. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models11 Mar

Other references · 37

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  9. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  10. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  11. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  12. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
25 more
  1. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  2. 2019Towards VQA Models That Can Readcited by 1,696
  3. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  4. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  5. 2020Rethinking Attention with Performerscited by 1,584
  6. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  7. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  8. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  9. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  10. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  11. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  12. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  13. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  14. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  15. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  16. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  17. 2018nocaps: novel object captioning at scalecited by 444
  18. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 344
  19. 2024TinyLLaVA: A Framework of Small-scale Large Multimodal Modelscited by 164
  20. 2019Adaptive Attention Span in Transformerscited by 148
  21. 2024VL-Mamba: Exploring State Space Models for Multimodal Learningcited by 93
  22. 2023TinyGPT-V: Efficient Multimodal Large Language Model via Small Backbonescited by 77
  23. 2024Attention Score is not All You Need for Token Importance Indicator in KV Cache Reduction: Value Also Matterscited by 73
  24. 2024LLAVADI: What Matters For Multimodal Large Language Models Distillationcited by 25
  25. 2024Skipping Computations in Multimodal LLMscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 48; 2026: 129.

177 papers.

Sort:

By area:

  1. 2025Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMscited by 92
  2. 2025Spatial Reasoning with Vision-Language Models in Ego-Centric Multi-View Scenescited by 62
  3. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 49
  4. 2025SpecPrune-VLA: Accelerating Vision-Language-Action Models via Action-Aware Self-Speculative Pruningcited by 39
  5. 2025MMTok: Multimodal Coverage Maximization for Efficient Inference of VLMscited by 29
  6. 2025LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMscited by 25
  7. 2025SCOPE: Saliency-Coverage Oriented Token Pruning for Efficient Multimodel LLMscited by 23
  8. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 22
  9. 2025A Glimpse to Compress: Dynamic Visual Token Pruning for Large Vision-Language Modelscited by 18
  10. 2025Less Is More, but Where? Dynamic Token Compression via LLM-Guided Keyframe Priorcited by 15
  11. 2025Bridging the Semantic-Action Gap in Visual Token Pruning for Efficient VLA Inferencecited by 14
  12. 2026FlashVID: Efficient Video Large Language Models via Training-free Tree-based Spatiotemporal Token Mergingcited by 14
38 more
  1. 2026N"uwa: Mending the Spatial Integrity Torn by VLM Token Pruningcited by 13
  2. 2025VisionSelector: End-to-End Learnable Visual Token Compression for Efficient Multimodal LLMscited by 11
  3. 2025Can Visual Input Be Compressed? A Visual Token Compression Benchmark for Large Multimodal Modelscited by 11
  4. 2025When Token Pruning is Worse than Random: Understanding Visual Token Information in VLLMscited by 11
  5. 2026The Model Knows Which Tokens Matter: Automatic Token Selection via Noise Gatingcited by 11
  6. 2026LLaVA-FA: Learning Fourier Approximation for Compressing Large Multimodal Modelscited by 10
  7. 2026Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Evictioncited by 10
  8. 2025StreamingAssistant: Efficient Visual Token Pruning for Accelerating Online Video Understandingcited by 9
  9. 2026Language-Guided Token Compression with Reinforcement Learning in Large Vision-Language Modelscited by 9
  10. 2026ViCA: Efficient Multimodal LLMs with Vision-Only Cross-Attentioncited by 8
  11. 2026ResidualKV: Residual-Based KV Cache Compression for Efficient Long-Context Inferencecited by 8
  12. 2025ToDRE: Effective Visual Token Pruning via Token Diversity and Task Relevancecited by 7
  13. 2025SalQ-VLM: Fine-Grained Saliency-Guided Quantization for Vision-Language Modelscited by 7
  14. 2025SLAM-Former: Putting SLAM into One Transformercited by 7
  15. 2025Compressor-VLA: Instruction-Guided Visual Token Compression for Efficient Robotic Manipulationcited by 7
  16. 2026AgilePruner: An Empirical Study of Attention and Diversity for Adaptive Visual Token Pruning in Large Vision-Language Modelscited by 7
  17. 2026LearnPruner: Rethinking Attention-based Token Pruning in Vision Language Modelscited by 7
  18. 2025Efficient Whole Slide Pathology VQA via Token Compressioncited by 6
  19. 2025FLoC: Facility Location-Based Efficient Visual Token Compression for Long Video Understandingcited by 6
  20. 2025LoC-Path: Learning to Compress for Pathology Multimodal Large Language Modelscited by 6
  21. 2026Unified Spatiotemporal Token Compression for Video-LLMs at Ultra-Low Retentioncited by 6
  22. 2026Photon: Speedup Volume Understanding with Efficient Multimodal Large Language Modelscited by 6
  23. 2026Why and When Visual Token Pruning Fails? A Study on Relevant Visual Information Shift in MLLMs Decodingcited by 6
  24. 2025VLM-Pruner: Buffering for Spatial Sparsity in an Efficient VLM Centrifugal Token Pruning Paradigmcited by 5
  25. 2026On the Adversarial Robustness of Large Vision-Language Models under Visual Token Compressioncited by 5
  26. 2026AngelSlim: A more accessible, comprehensive, and efficient toolkit for large model compressioncited by 5
  27. 2026EvoPrune: Early-Stage Visual Token Pruning for Efficient MLLMscited by 5
  28. 2026Improving Visual Token Reduction via Rectifying Distortions for Efficient Multimodal LLM Inferencecited by 5
  29. 2025SpecVLM: Fast Speculative Decoding in Vision-Language Modelscited by 4
  30. 2025ZOO-Prune: Training-Free Token Pruning via Zeroth-Order Gradient Estimation in Vision-Language Modelscited by 4
  31. 2025TimeViper: A Hybrid Mamba-Transformer Vision-Language Model for Efficient Long Video Understandingcited by 4
  32. 2026ForestPrune: High-ratio Visual Token Compression for Video Multimodal Large Language Models via Spatial-Temporal Forest Modelingcited by 4
  33. 2026DocPrune:Efficient Document Question Answering via Background, Question, and Comprehension-aware Token Pruningcited by 4
  34. 2026RTPrune: Reading-Twice Inspired Token Pruning for Efficient DeepSeek-OCR Inferencecited by 4
  35. 2026SAFE-Pruner: Semantic Attention-Guided Future-Aware Token Pruning for Efficient Vision-Language-Action Manipulationcited by 4
  36. 2026AVIS: Adaptive Test-Time Scaling for Vision-Language Modelscited by 4
  37. 2026Towards Fast and Effective Long Video Understanding of Multimodal Large Language Models via Adaptive Quasi-Gaussian Samplingcited by 4
  38. 2025TransPrune: Token Transition Pruning for Efficient Large Vision-Language Modelcited by 3

50 most cited of 177 All 177 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025TrimTokenator-LC: Towards Adaptive Visual Token Pruning for Large Multimodal Models with Long Contexts
  2. 2025TrimTokenator: Towards Adaptive Visual Token Pruning for Large Multimodal Models
  3. 2026SinkPruner: Sink-Free Visual Token Pruning for Multimodal Large Language Models
  4. 2024Sparsity Meets Similarity: Leveraging Long-Tail Distribution for Dynamic Optimized Token Representation in Multimodal Large Language Models
  5. 2026IDPruner: Harmonizing Importance and Diversity in Visual Token Pruning for MLLMs
  6. 2025TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Model
  7. 2025VFlowOpt: A Token Pruning Framework for LMMs with Visual Information Flow-Guided Optimization
  8. 2025What Kind of Visual Tokens Do We Need? Training-free Visual Token Pruning for Multi-modal Large Language Models from the Perspective of Graph