mlsanityData to 7 Oct 2026

Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language Models

arXiv 2409.10197 ↗First posted 16 Sep 2024cs.CV cs.CL cs.MMcited by 125

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

also in 1 more
Abstract (arXiv)

Recent progress in Multimodal Large Language Models(MLLMs) often use large image tokens to compensate the visual shortcoming of MLLMs, which not only exhibits obvious redundancy but also greatly exacerbates the already high computation. Token pruning is an effective solution for speeding up MLLMs, but when and how to drop tokens still remains a challenge. In this paper, we propose a novel and training-free approach for the effective visual token pruning of MLLMs, termed FitPrune, which can quickly produce a complete pruning recipe for MLLMs according to a pre-defined budget. Specifically, FitPrune considers token pruning as a statistical problem of MLLM and its objective is to find out an optimal pruning scheme that can minimize the divergence of the attention distributions before and after pruning. In practice, FitPrune can be quickly accomplished based on the attention statistics from a small batch of inference data, avoiding the expensive trials of MLLMs. According to the pruning recipe, an MLLM can directly remove the redundant visual tokens of different examples during inference. To validate FitPrune, we apply it to a set of recent MLLMs, including LLaVA-1.5, LLaVA-HR and LLaVA-NEXT, and conduct extensive experiments on a set of benchmarks. The experimental results show that our FitPrune can not only reduce the computational complexity to a large extent, while retaining high performance, e.g., -54.9% FLOPs for LLaVA-NEXT with only 0.5% accuracy drop. Notably, the pruning recipe can be obtained in about 5 minutes. Our code is available at https://github.com/ywh187/FitPrune.

Key facts. Cited by 125 papers in the map (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 25 of its 41 references that are in the map. First posted on arXiv 16 Sep 2024.

Builds on

25 of 41 references.

References · 25

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  9. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  10. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  11. 2019Towards VQA Models That Can Readcited by 1,696
  12. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
13 more
  1. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  2. 2022Token Merging: Your ViT But Fastercited by 995
  3. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  4. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  5. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  6. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  7. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  8. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruningcited by 165
  9. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
  10. 2023Joint Token Pruning and Squeezing Towards More Aggressive Compression of Vision Transformerscited by 79
  11. 2023Zero-TPrune: Zero-Shot Token Pruning through Leveraging of the Attention Graph in Pre-Trained Transformerscited by 61
  12. 2022HeatViT: Hardware-Efficient Adaptive Token Pruning for Vision Transformerscited by 46
  13. 2021TR-BERT: Dynamic Token Reduction for Accelerating BERT Inferencecited by 40

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 42; 2026: 76.

125 papers.

Sort:

By area:

  1. 2025DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Modelscited by 177
  2. 2024Beyond Text-Visual Attention: Exploiting Visual Cues for Effective Token Pruning in VLMscited by 153
  3. 2025Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMscited by 92
  4. 2025MoLe-VLA: Dynamic Layer-skipping Vision Language Action Model via Mixture-of-Layers for Efficient Robot Manipulationcited by 69
  5. 2025Don't Just Chase "Highlighted Tokens" in MLLMs: Revisiting Visual Holistic Context Retentioncited by 45
  6. 2024AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruningcited by 43
  7. 2025SpecPrune-VLA: Accelerating Vision-Language-Action Models via Action-Aware Self-Speculative Pruningcited by 39
  8. 2025When Large Vision-Language Model Meets Large Remote Sensing Imagery: Coarse-to-Fine Text-Guided Token Pruningcited by 35
  9. 2024ReTaKe: Reducing Temporal and Knowledge Redundancy for Long Video Understandingcited by 33
  10. 2025Global Compression Commander: Plug-and-Play Inference Acceleration for High-Resolution Large Vision-Language Modelscited by 32
  11. 2026OmniSIFT: Modality-Asymmetric Token Compression for Efficient Omni-modal Large Language Modelscited by 23
  12. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 22
38 more
  1. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 22
  2. 2025VFlowOpt: A Token Pruning Framework for LMMs with Visual Information Flow-Guided Optimizationcited by 21
  3. 2025OmniZip: Audio-Guided Dynamic Token Compression for Fast Omnimodal Large Language Modelscited by 21
  4. 2025DocPruner: A Storage-Efficient Framework for Multi-Vector Visual Document Retrieval via Adaptive Patch-Level Embedding Pruningcited by 19
  5. 2024Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findingscited by 15
  6. 2025Growing a Multi-head Twig via Distillation and Reinforcement Learning to Accelerate Large Vision-Language Modelscited by 15
  7. 2025SQAP-VLA: A Synergistic Quantization-Aware Pruning Framework for High-Performance Vision-Language-Action Modelscited by 15
  8. 2025VisiPruner: Decoding Discontinuous Cross-Modal Dynamics for Efficient Multimodal LLMscited by 14
  9. 2025Bridging the Semantic-Action Gap in Visual Token Pruning for Efficient VLA Inferencecited by 14
  10. 2025CrossLMM: Decoupling Long Video Sequences from LMMs via Dual Cross-Attention Mechanismscited by 11
  11. 2025Can Visual Input Be Compressed? A Visual Token Compression Benchmark for Large Multimodal Modelscited by 11
  12. 2025When Token Pruning is Worse than Random: Understanding Visual Token Information in VLLMscited by 11
  13. 2026IVC-Prune: Revealing the Implicit Visual Coordinates in LVLMs for Vision Token Pruningcited by 11
  14. 2026The Model Knows Which Tokens Matter: Automatic Token Selection via Noise Gatingcited by 11
  15. 2024FlashSloth: Lightning Multimodal Large Language Models via Embedded Visual Compressioncited by 10
  16. 2025Why 1 + 1 < 1 in Visual Token Pruning: Beyond Naive Integration via Multi-Objective Balanced Coveringcited by 10
  17. 2026ViCA: Efficient Multimodal LLMs with Vision-Only Cross-Attentioncited by 8
  18. 2025Beyond Intermediate States: Explaining Visual Redundancy through Languagecited by 7
  19. 2025ToDRE: Effective Visual Token Pruning via Token Diversity and Task Relevancecited by 7
  20. 2025Interpreting Attention Heads for Image-to-Text Information Flow in Large Vision-Language Modelscited by 7
  21. 2025Recurrent Attention-based Token Selection for Efficient Streaming Video-LLMscited by 7
  22. 2025FlowMM: Cross-Modal Information Flow Guided KV Cache Merging for Efficient Multimodal Context Inferencecited by 6
  23. 2025LoC-Path: Learning to Compress for Pathology Multimodal Large Language Modelscited by 6
  24. 2025Token Expand-Merge: Training-Free Token Compression for Vision-Language-Action Modelscited by 6
  25. 2026XStreamVGGT: Extremely Memory-Efficient Streaming Vision Geometry Grounded Transformer with KV Cache Compressioncited by 6
  26. 2026DASH: Dynamic Audio-Driven Semantic Chunking for Efficient Omnimodal Token Compressioncited by 6
  27. 2026StreamingVLA: Streaming Vision-Language-Action Model with Action Flow Matching and Adaptive Early Observationcited by 6
  28. 2026Toward Native Multimodal Modeling: A Roadmapcited by 6
  29. 2025Adaptive Token Merging for Efficient Transformer Semantic Communication at the Edgecited by 5
  30. 2026Improving Visual Token Reduction via Rectifying Distortions for Efficient Multimodal LLM Inferencecited by 5
  31. 2024Pruning All-Rounder: Rethinking and Improving Inference Efficiency for Large Vision Language Modelscited by 4
  32. 2024Deploying Foundation Model Powered Agent Services: A Surveycited by 4
  33. 2025AdaFV: Rethinking of Visual-Language alignment for VLM accelerationcited by 4
  34. 2025SpecVLM: Fast Speculative Decoding in Vision-Language Modelscited by 4
  35. 2025ZOO-Prune: Training-Free Token Pruning via Zeroth-Order Gradient Estimation in Vision-Language Modelscited by 4
  36. 2026ForestPrune: High-ratio Visual Token Compression for Video Multimodal Large Language Models via Spatial-Temporal Forest Modelingcited by 4
  37. 2026RTPrune: Reading-Twice Inspired Token Pruning for Efficient DeepSeek-OCR Inferencecited by 4
  38. 2026Towards a Dynamic and Fixed-budget Memory Bank for Efficient Streaming Video Understandingcited by 4

50 most cited of 125 All 125 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026SinkPruner: Sink-Free Visual Token Pruning for Multimodal Large Language Models
  2. 2026Fast-Slow Efficient Training for Multimodal Large Language Models via Visual Token Pruning
  3. 2026S2Prune: Spatially Structured Visual Token Pruning for Multimodal Large Language Models
  4. 2026IDPruner: Harmonizing Importance and Diversity in Visual Token Pruning for MLLMs
  5. 2026EvoPrune: Early-Stage Visual Token Pruning for Efficient MLLMs
  6. 2025What Kind of Visual Tokens Do We Need? Training-free Visual Token Pruning for Multi-modal Large Language Models from the Perspective of Graph
  7. 2026FastAV: Efficient Token Pruning for Audio-Visual Large Language Model Inference
  8. 2026Capturing Token Tendencies for Training-Free Token Pruning in Multimodal Large Language Models