mlsanityData to 13 Oct 2026

Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference

arXiv 2405.05803 ↗First posted 9 May 2024cs.CV cs.AIcited by 137

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

Multimodal large language models (MLLMs) demand considerable computations for inference due to the extensive parameters and the additional input tokens needed for visual information representation. Herein, we introduce Visual Tokens Withdrawal (VTW), a plug-and-play module to boost MLLMs for rapid inference. Our approach is inspired by two intriguing phenomena we have observed: (1) the attention sink phenomenon that is prevalent in LLMs also persists in MLLMs, suggesting that initial tokens and nearest tokens receive the majority of attention, while middle vision tokens garner minimal attention in deep layers; (2) the presence of information migration, which implies that visual information is transferred to subsequent text tokens within the first few layers of MLLMs. As per our findings, we conclude that vision tokens are unnecessary in the deep layers of MLLMs. Thus, we strategically withdraw them at a certain layer, enabling only text tokens to engage in subsequent layers. To pinpoint the ideal layer for VTW, we initially analyze a limited set of tiny datasets and choose the first layer that meets the Kullback-Leibler divergence criterion. Our VTW approach can cut computational overhead by over 40% across diverse multimodal tasks while maintaining performance.

Key facts. Cited by 137 papers in the map (data through 13 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 30 of its 48 references that are in the map. First posted on arXiv 9 May 2024.

Builds on

30 of 48 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models22 Mar
  2. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models11 Mar
  3. 2023CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformers27 May

Other references · 27

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
  3. 2022Training language models to follow instructions with human feedbackcited by 15,856
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,979
  6. 2023Visual Instruction Tuningcited by 7,827
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,103
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,389
  9. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,637
  10. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,546
  11. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,204
  12. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,137
15 more
  1. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,057
  2. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,896
  3. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,709
  4. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,465
  5. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,392
  6. 2022Token Merging: Your ViT But Fastercited by 1,000
  7. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 852
  8. 2016A Diagram Is Worth A Dozen Imagescited by 796
  9. 2022Efficiently Scaling Transformer Inferencecited by 532
  10. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  11. 2021Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformercited by 175
  12. 2021Learned Token Pruning for Transformerscited by 139
  13. 2023DiffRate : Differentiable Compression Rate for Efficient Vision Transformerscited by 77
  14. 2021Token Pooling in Vision Transformerscited by 68
  15. 2022CF-ViT: A General Coarse-to-Fine Method for Vision Transformercited by 63

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 11; 2025: 61; 2026: 65.

137 papers.

Sort:

By area:

  1. 2025DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Modelscited by 177
  2. 2024Efficient Multimodal Large Language Models: A Surveycited by 95
  3. 2025TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Modelcited by 57
  4. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 50
  5. 2025FastVID: Dynamic Density Pruning for Fast Video Large Language Modelscited by 49
  6. 2025PACT: Pruning and Clustering-Based Token Reduction for Faster Visual Language Modelscited by 49
  7. 2024MBQ: Modality-Balanced Quantization for Large Vision-Language Modelscited by 45
  8. 2024AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruningcited by 43
  9. 2025VScan: Rethinking Visual Token Reduction for Efficient Large Vision-Language Modelscited by 41
  10. 2024DocKylin: A Large Multimodal Model for Visual Document Understanding with Efficient Visual Slimmingcited by 35
  11. 2025What Kind of Visual Tokens Do We Need? Training-free Visual Token Pruning for Multi-modal Large Language Models from the Perspective of Graphcited by 35
  12. 2025Harnessing the Reasoning Economy: A Survey of Efficient Reasoning for Large Language Modelscited by 35
38 more
  1. 2025Don't Blind Your VLA: Aligning Visual Representations for OOD Generalizationcited by 34
  2. 2025Global Compression Commander: Plug-and-Play Inference Acceleration for High-Resolution Large Vision-Language Modelscited by 32
  3. 2024Accelerating Multimodal Large Language Models by Searching Optimal Vision Token Reductioncited by 28
  4. 2024LLAVADI: What Matters For Multimodal Large Language Models Distillationcited by 26
  5. 2025LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMscited by 25
  6. 2025Lifting the Veil on Visual Information Flow in MLLMs: Unlocking Pathways to Faster Inferencecited by 23
  7. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 22
  8. 2024FocusLLaVA: A Coarse-to-Fine Approach for Efficient and Effective Visual Token Compressioncited by 21
  9. 2024ST3: Accelerating Multimodal Large Language Model by Spatial-Temporal Visual Token Trimmingcited by 20
  10. 2025Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimizationcited by 20
  11. 2025ModServe: Modality- and Stage-Aware Resource Disaggregation for Scalable Multimodal Model Servingcited by 18
  12. 2025CATP: Contextually Adaptive Token Pruning for Efficient and Enhanced Multimodal In-Context Learningcited by 18
  13. 2024Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findingscited by 15
  14. 2025Bridging the Semantic-Action Gap in Visual Token Pruning for Efficient VLA Inferencecited by 15
  15. 2025TrimTokenator-LC: Towards Adaptive Visual Token Pruning for Large Multimodal Models with Long Contextscited by 14
  16. 2025ShortV: Efficient Multimodal Large Language Models by Freezing Visual Tokens in Ineffective Layerscited by 13
  17. 2025Constructive Distortion: Improving MLLMs with Attention-Guided Image Warpingcited by 13
  18. 2025Can Visual Input Be Compressed? A Visual Token Compression Benchmark for Large Multimodal Modelscited by 11
  19. 2025When Token Pruning is Worse than Random: Understanding Visual Token Information in VLLMscited by 11
  20. 2026The Model Knows Which Tokens Matter: Automatic Token Selection via Noise Gatingcited by 11
  21. 2024AVG-LLaVA: An Efficient Large Multimodal Model with Adaptive Visual Granularitycited by 9
  22. 2024Elevating Visual Perception in Multimodal LLMs with Visual Embedding Distillationcited by 9
  23. 2025TrimTokenator: Towards Adaptive Visual Token Pruning for Large Multimodal Modelscited by 9
  24. 2025Widget2Code: From Visual Widgets to UI Code via Multimodal LLMscited by 8
  25. 2026ViCA: Efficient Multimodal LLMs with Vision-Only Cross-Attentioncited by 8
  26. 2025ToDRE: Effective Visual Token Pruning via Token Diversity and Task Relevancecited by 7
  27. 2025ElasticMM: Efficient Multimodal LLMs Serving with Elastic Multimodal Parallelismcited by 7
  28. 2025Short-LVLM: Compressing and Accelerating Large Vision-Language Models by Pruning Redundant Layerscited by 7
  29. 2025RedundancyLens: Revealing and Exploiting Visual Token Processing Redundancy for Efficient Decoder-Only MLLMscited by 6
  30. 2025PLPHP: Per-Layer Per-Head Vision Token Pruning for Efficient Large Vision-Language Modelscited by 6
  31. 2025IKOD: Mitigating Visual Attention Degradation in Large Vision-Language Modelscited by 6
  32. 2025Investigating Redundancy in Multimodal Large Language Models with Multiple Vision Encoderscited by 5
  33. 2025VISA: Group-wise Visual Token Selection and Aggregation via Graph Summarization for Efficient MLLMs Inferencecited by 5
  34. 2025CARES: Context-Aware Resolution Selector for VLMscited by 5
  35. 2025LinkedOut: Linking World Knowledge Representation Out of Video LLM for Next-Generation Video Recommendationcited by 5
  36. 2026FastAV: Efficient Token Pruning for Audio-Visual Large Language Model Inferencecited by 5
  37. 2026ScalSelect: Scalable Training-Free Multimodal Data Selection for Efficient Visual Instruction Tuningcited by 5
  38. 2026Test-Time Attention Purification for Backdoored Large Vision Language Modelscited by 5

50 most cited of 137 All 137 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMs
  2. 2026Fast-Slow Efficient Training for Multimodal Large Language Models via Visual Token Pruning
  3. 2026Stepwise Token Selection for Efficient Multimodal Large Language Models
  4. 2024Treat Visual Tokens as Text? But Your MLLM Only Needs Fewer Efforts to See
  5. 2025LFTR: Learning-Free Token Reduction for Multimodal Large Language Models
  6. 2026What Do Visual Tokens Really Encode? Uncovering Sparsity and Redundancy in Multimodal Large Language Models
  7. 2026Capturing Token Tendencies for Training-Free Token Pruning in Multimodal Large Language Models
  8. 2026How Many Visual Tokens Do Multimodal Language Models Need? Scaling Visual Token Pruning with F^3A