mlsanityData to 7 Oct 2026

Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs

arXiv 2506.10967 ↗First posted 12 Jun 2025cs.CV cs.AIcited by 92

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

also in 1 more
Abstract (arXiv)

In multimodal large language models (MLLMs), the length of input visual tokens is often significantly greater than that of their textual counterparts, leading to a high inference cost. Many works aim to address this issue by removing redundant visual tokens. However, current approaches either rely on attention-based pruning, which retains numerous duplicate tokens, or use similarity-based pruning, overlooking the instruction relevance, consequently causing suboptimal performance. In this paper, we go beyond attention or similarity by proposing a novel visual token pruning method named CDPruner, which maximizes the conditional diversity of retained tokens. We first define the conditional similarity between visual tokens conditioned on the instruction, and then reformulate the token pruning problem with determinantal point process (DPP) to maximize the conditional diversity of the selected subset. The proposed CDPruner is training-free and model-agnostic, allowing easy application to various MLLMs. Extensive experiments across diverse MLLMs show that CDPruner establishes new state-of-the-art on various vision-language benchmarks. By maximizing conditional diversity through DPP, the selected subset better represents the input images while closely adhering to user instructions, thereby preserving strong performance even with high reduction ratios. When applied to LLaVA, CDPruner reduces FLOPs by 95% and CUDA latency by 78%, while maintaining 94% of the original accuracy. Our code is available at https://github.com/Theia-4869/CDPruner.

Key facts. Cited by 92 papers in the map (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 64 of its 84 references that are in the map. First posted on arXiv 12 Jun 2025.

Builds on

64 of 84 references.

Closest ancestors · 17

Same direction, newest first

  1. 2025TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Model24 Mar
  2. 2025Similarity-Aware Token Pruning: Your VLM but Faster14 Mar
  3. 2025DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Models4 Mar
  4. 2025Token Pruning in Multimodal Large Language Models: Are We Solving the Right Problem?17 Feb
  5. 2025Stop Looking for Important Tokens in Multimodal Language Models: Duplication Matters More17 Feb
  6. 2025LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token7 Jan
  7. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Models5 Dec
  8. 2024Beyond Text-Visual Attention: Exploiting Visual Cues for Effective Token Pruning in VLMs2 Dec
  9. 2024Multi-Stage Vision Token Dropping: Towards Efficient Multimodal Large Language Model16 Nov
  10. 2024PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reduction22 Oct
  11. 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference6 Oct
  12. 2024Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language Models16 Sep
5 more
  1. 2024Matryoshka Query Transformer for Large Vision-Language Models29 May
  2. 2024Matryoshka Multimodal Models27 May
  3. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models22 Mar
  4. 2024LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Images18 Mar
  5. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models11 Mar

Other references · 47

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2025Qwen2.5-VL Technical Reportcited by 5,418
  8. 2024Qwen2.5 Technical Reportcited by 4,713
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  10. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  11. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  12. 2023Qwen Technical Reportcited by 3,273
35 more
  1. 2023Mistral 7Bcited by 3,065
  2. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  3. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  4. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  5. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  7. 2015VQA: Visual Question Answeringcited by 2,489
  8. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  9. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  10. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  11. 2019Towards VQA Models That Can Readcited by 1,696
  12. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  13. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  14. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  15. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  16. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  17. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  18. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  19. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  20. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  21. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  22. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  23. 2016A Diagram Is Worth A Dozen Imagescited by 793
  24. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  25. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 644
  26. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  27. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  28. 2024Long Context Transfer from Language to Visioncited by 502
  29. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 466
  30. 2024InternLM2 Technical Reportcited by 391
  31. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 303
  32. 2024Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language Modelscited by 112
  33. 2024MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMscited by 89
  34. 2025MDP3: A Training-free Approach for List-wise Frame Selection in Video-LLMscited by 40
  35. 2018Fair and Diverse DPP-based Data Summarizationcited by 39

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 16; 2026: 76.

92 papers.

Sort:

By area:

  1. 2025The Better You Learn, The Smarter You Prune: Towards Efficient Vision-language-action Models via Differentiable Token Pruningcited by 34
  2. 2025TIGeR: Tool-Integrated Geometric Reasoning in Vision-Language Models for Roboticscited by 27
  3. 2025FastDriveVLA: Efficient End-to-End Driving via Plug-and-Play Reconstruction-based Token Pruningcited by 20
  4. 2025A Glimpse to Compress: Dynamic Visual Token Pruning for Large Vision-Language Modelscited by 18
  5. 2025CATP: Contextually Adaptive Token Pruning for Efficient and Enhanced Multimodal In-Context Learningcited by 18
  6. 2025TrimTokenator-LC: Towards Adaptive Visual Token Pruning for Large Multimodal Models with Long Contextscited by 14
  7. 2025Rethinking Visual Token Reduction in LVLMs Under Cross-Modal Misalignmentcited by 11
  8. 2025When Token Pruning is Worse than Random: Understanding Visual Token Information in VLLMscited by 11
  9. 2025HiPrune: Hierarchical Attention for Efficient Token Pruning in Vision-Language Modelscited by 10
  10. 2026Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Evictioncited by 10
  11. 2025TrimTokenator: Towards Adaptive Visual Token Pruning for Large Multimodal Modelscited by 9
  12. 2026ViCA: Efficient Multimodal LLMs with Vision-Only Cross-Attentioncited by 8
38 more
  1. 2026Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospectscited by 8
  2. 2026SaPaVe: Towards Active Perception and Manipulation in Vision-Language-Action Models for Roboticscited by 6
  3. 2026Photon: Speedup Volume Understanding with Efficient Multimodal Large Language Modelscited by 6
  4. 2025VLM-Pruner: Buffering for Spatial Sparsity in an Efficient VLM Centrifugal Token Pruning Paradigmcited by 5
  5. 2026AngelSlim: A more accessible, comprehensive, and efficient toolkit for large model compressioncited by 5
  6. 2026EvoPrune: Early-Stage Visual Token Pruning for Efficient MLLMscited by 5
  7. 2025EgoPrune: Efficient Token Pruning for Egomotion Video Reasoning in Embodied Agentcited by 4
  8. 2025MathSticks: A Benchmark for Visual Symbolic Compositional Reasoning with Matchstick Puzzlescited by 4
  9. 2026IDPruner: Harmonizing Importance and Diversity in Visual Token Pruning for MLLMscited by 4
  10. 2026RTPrune: Reading-Twice Inspired Token Pruning for Efficient DeepSeek-OCR Inferencecited by 4
  11. 2026SAFE-Pruner: Semantic Attention-Guided Future-Aware Token Pruning for Efficient Vision-Language-Action Manipulationcited by 4
  12. 2026AVIS: Adaptive Test-Time Scaling for Vision-Language Modelscited by 4
  13. 2025TransPrune: Token Transition Pruning for Efficient Large Vision-Language Modelcited by 3
  14. 2025Attention Debiasing for Token Pruning in Vision Language Modelscited by 3
  15. 2026Focus-Scan-Refine: From Human Visual Perception to Efficient Visual Token Pruningcited by 3
  16. 2026Stateful Token Reduction for Long-Video Hybrid VLMscited by 3
  17. 2026HAWK: Head Importance-Aware Visual Token Pruning in Multimodal Modelscited by 3
  18. 2026OccamToken: Efficient VLM Inference with Training-Free and Budget-Adaptive Token Pruningcited by 3
  19. 2026Moving Beyond Diversity: Visual Token Pruning as Subspace Reconstruction for Efficient VLMscited by 3
  20. 2026AnchorPrune: Relevance-Anchored Contextual Expansion for Visual Token Pruningcited by 3
  21. 2026HieraVid: Hierarchical Token Pruning for Fast Video Large Language Modelscited by 2
  22. 2026MI-Pruner: Crossmodal Mutual Information-guided Token Pruner for Efficient MLLMscited by 2
  23. 2026LDDR: Linear-DPP-Based Dynamic-Resolution Frame Sampling for Video MLLMscited by 2
  24. 2026TOPS: First-Principles Visual Token Pruning via Constructing Token Optimal Preservation Sets for Efficient MLLM Inferencecited by 2
  25. 2026ERA: Entropy-Guided Visual Token Pruning with Rectified Attention for Efficient MLLMscited by 2
  26. 2026WaveZip: Wavelet-Driven Space-Time Decoupling for Video Token Condensationcited by 2
  27. 2026PathSelect: Sequential Token Selection for Whole Slide Pathologycited by 2
  28. 2026Who Speaks for the Pruned? Visual Token Pruning as Coverage Optimizationcited by 2
  29. 2025Fine-grained Token Allocation Via Operation Pruning for Efficient MLLMscited by 1
  30. 2026TRIO: Token Reduction via Inference-Objective Guidance for Efficient Vision-Language Modelscited by 1
  31. 2026Frequency-Modulated Visual Restoration for Matryoshka Large Multimodal Modelscited by 1
  32. 2026ReDiPrune: Relevance-Diversity Pre-Projection Token Pruning for Efficient Multimodal LLMscited by 1
  33. 2026QAPruner: Quantization-Aware Vision Token Pruning for Multimodal Large Language Modelscited by 1
  34. 2026When Does Visual Token Pruning Improve Calibration? The Role of Evidence Coverage in MLLMscited by 1
  35. 2026ERASE: Eliminating Redundant Visual Tokens via Adaptive Two-Stage Token Pruningcited by 1
  36. 2026EchoPrune: Interpreting Redundancy as Temporal Echoes for Efficient VideoLLMscited by 1
  37. 2026How Many Visual Tokens Do Multimodal Language Models Need? Scaling Visual Token Pruning with F^3Acited by 1
  38. 2026TGV-KV: Text-Grounded KV Eviction for Vision-Language Modelscited by 1

50 most cited of 92 All 92 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Don't Just Chase "Highlighted Tokens" in MLLMs: Revisiting Visual Holistic Context Retention
  2. 2026Capturing Token Tendencies for Training-Free Token Pruning in Multimodal Large Language Models
  3. 2025Script: Graph-Structured and Query-Conditioned Semantic Token Pruning for Multimodal Large Language Models
  4. 2026SinkPruner: Sink-Free Visual Token Pruning for Multimodal Large Language Models
  5. 2025What Kind of Visual Tokens Do We Need? Training-free Visual Token Pruning for Multi-modal Large Language Models from the Perspective of Graph
  6. 2025ToDRE: Effective Visual Token Pruning via Token Diversity and Task Relevance
  7. 2026Multi-Image Visual Token Pruning in Large Visual Language Models
  8. 2024ST3: Accelerating Multimodal Large Language Model by Spatial-Temporal Visual Token Trimming