mlsanityData to 7 Oct 2026

VisionZip: Longer is Better but Not Necessary in Vision Language Models

arXiv 2412.04467 ↗First posted 5 Dec 2024cs.CV cs.AI cs.CLcited by 298

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

Recent advancements in vision-language models have enhanced performance by increasing the length of visual tokens, making them much longer than text tokens and significantly raising computational costs. However, we observe that the visual tokens generated by popular vision encoders, such as CLIP and SigLIP, contain significant redundancy. To address this, we introduce VisionZip, a simple yet effective method that selects a set of informative tokens for input to the language model, reducing visual token redundancy and improving efficiency while maintaining model performance. The proposed VisionZip can be widely applied to image and video understanding tasks and is well-suited for multi-turn dialogues in real-world scenarios, where previous methods tend to underperform. Experimental results show that VisionZip outperforms the previous state-of-the-art method by at least 5% performance gains across nearly all settings. Moreover, our method significantly enhances model inference speed, improving the prefilling time by 8x and enabling the LLaVA-Next 13B model to infer faster than the LLaVA-Next 7B model while achieving better results. Furthermore, we analyze the causes of this redundancy and encourage the community to focus on extracting better visual features rather than merely increasing token length. Our code is available at https://github.com/dvlab-research/VisionZip .

Key facts. Cited by 298 papers in the map (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 65 of its 79 references that are in the map. First posted on arXiv 5 Dec 2024.

Builds on

65 of 79 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reduction22 Oct
  2. 2024Efficient Vision-Language Models by Summarizing Visual Tokens into Compact Registers17 Oct
  3. 2024ZipVL: Efficient Large Vision-Language Models with Dynamic Token Sparsification11 Oct
  4. 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference6 Oct
  5. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models22 Mar
  6. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models11 Mar

Other references · 59

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  10. 2022A ConvNet for the 2020scited by 3,545
  11. 2023Qwen Technical Reportcited by 3,273
  12. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
38 more
  1. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  2. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  3. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  4. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  6. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  7. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  8. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  9. 2019Towards VQA Models That Can Readcited by 1,696
  10. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  11. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  12. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  13. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  14. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  15. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  16. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  17. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  18. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  19. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  20. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  21. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  22. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  23. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  24. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  25. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  26. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  27. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  28. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 366
  29. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  30. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 344
  31. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 303
  32. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 265
  33. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 243
  34. 2023Video Understanding with Large Language Models: A Surveycited by 203
  35. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  36. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
  37. 2024LITA: Language Instructed Temporal-Localization Assistantcited by 124
  38. 2023LISA++: An Improved Baseline for Reasoning Segmentation with Large Language Modelcited by 70

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 107; 2026: 187.

298 papers.

Sort:

By area:

  1. 2025Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMscited by 92
  2. 2024FastVLM: Efficient Vision Encoding for Vision Language Modelscited by 76
  3. 2025MedAgent-Pro: Towards Evidence-based Multi-modal Medical Diagnosis via Reasoning Agentic Workflowcited by 71
  4. 2025HoliTom: Holistic Token Merging for Fast Video Large Language Modelscited by 58
  5. 2025SP-VLA: A Joint Model Scheduling and Token Pruning Approach for VLA Model Accelerationcited by 56
  6. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 49
  7. 2025FastVID: Dynamic Density Pruning for Fast Video Large Language Modelscited by 47
  8. 2025Don't Just Chase "Highlighted Tokens" in MLLMs: Revisiting Visual Holistic Context Retentioncited by 45
  9. 2026Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipecited by 43
  10. 2025VScan: Rethinking Visual Token Reduction for Efficient Large Vision-Language Modelscited by 42
  11. 2025VisionThink: Smart and Efficient Vision Language Model via Reinforcement Learningcited by 41
  12. 2025StreamingTOM: Streaming Token Compression for Efficient Video Understandingcited by 41
38 more
  1. 2024Feather the Throttle: Revisiting Visual Token Pruning for Vision-Language Model Accelerationcited by 40
  2. 2025Visual Representation Alignment for Multimodal Large Language Modelscited by 40
  3. 2025Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmarkcited by 37
  4. 2025When Large Vision-Language Model Meets Large Remote Sensing Imagery: Coarse-to-Fine Text-Guided Token Pruningcited by 35
  5. 2025Accelerating Streaming Video Large Language Models via Hierarchical Token Compressioncited by 34
  6. 2025Global Compression Commander: Plug-and-Play Inference Acceleration for High-Resolution Large Vision-Language Modelscited by 32
  7. 2025Similarity-Aware Token Pruning: Your VLM but Fastercited by 32
  8. 2025FOLDER: Accelerating Multi-modal Large Language Models with Enhanced Performancecited by 31
  9. 2025Think Twice, Act Once: Token-Aware Compression and Action Reuse for Efficient Inference in Vision-Language-Action Modelscited by 30
  10. 2025Shifting AI Efficiency From Model-Centric to Data-Centric Compressioncited by 29
  11. 2025FlowCut: Rethinking Redundancy via Information Flow for Efficient Vision-Language Modelscited by 29
  12. 2025MMTok: Multimodal Coverage Maximization for Efficient Inference of VLMscited by 29
  13. 2025PixelThink: Towards Efficient Chain-of-Pixel Reasoningcited by 28
  14. 2025Mitigating Object Hallucinations via Sentence-Level Early Interventioncited by 28
  15. 2025Scaling Vision Pre-Training to 4K Resolutioncited by 26
  16. 2025MMInference: Accelerating Pre-filling for Long-Context VLMs via Modality-Aware Permutation Sparse Attentioncited by 26
  17. 2026FluxMem: Adaptive Hierarchical Memory for Streaming Video Understandingcited by 26
  18. 2025LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMscited by 25
  19. 2025Variation-aware Vision Token Dropping for Faster Large Vision-Language Modelscited by 25
  20. 2025SCOPE: Saliency-Coverage Oriented Token Pruning for Efficient Multimodel LLMscited by 23
  21. 2026OmniSIFT: Modality-Asymmetric Token Compression for Efficient Omni-modal Large Language Modelscited by 23
  22. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 22
  23. 2025VFlowOpt: A Token Pruning Framework for LMMs with Visual Information Flow-Guided Optimizationcited by 21
  24. 2025OmniZip: Audio-Guided Dynamic Token Compression for Fast Omnimodal Large Language Modelscited by 21
  25. 2024Lyra: An Efficient and Speech-Centric Framework for Omni-Cognitioncited by 20
  26. 2025CT-Agent: A Multimodal-LLM Agent for 3D CT Radiology Question Answeringcited by 20
  27. 2025Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimizationcited by 20
  28. 2025TokenCarve: Information-Preserving Visual Token Compression in Multimodal Large Language Modelscited by 19
  29. 2025Does Your Vision-Language Model Get Lost in the Long Video Sampling Dilemma?cited by 19
  30. 2025OBS-Diff: Accurate Pruning For Diffusion Models in One-Shotcited by 19
  31. 2025AdaptVision: Efficient Vision-Language Models via Adaptive Visual Acquisitioncited by 19
  32. 2025Task-Oriented Feature Compression for Multimodal Understanding via Device-Edge Co-Inferencecited by 18
  33. 2025A Glimpse to Compress: Dynamic Visual Token Pruning for Large Vision-Language Modelscited by 18
  34. 2025FlexSelect: Flexible Token Selection for Efficient Long Video Understandingcited by 17
  35. 2026VisionTrim: Unified Vision Token Compression for Training-Free MLLM Accelerationcited by 16
  36. 2025Growing a Multi-head Twig via Distillation and Reinforcement Learning to Accelerate Large Vision-Language Modelscited by 15
  37. 2025SparseVILA: Decoupling Visual Sparsity for Efficient VLM Inferencecited by 15
  38. 2025Less Is More, but Where? Dynamic Token Compression via LLM-Guided Keyframe Priorcited by 15

50 most cited of 298 All 298 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospects
  2. 2025LLMC+: Benchmarking Vision-Language Model Compression with a Plug-and-play Toolkit
  3. 2025ZSPAPrune: Zero-Shot Prompt-Aware Token Pruning for Vision-Language Models
  4. 2026Can Vision-Language Models Handle Long-Context Code? An Empirical Study on Visual Compression
  5. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features
  6. 2024Inference Optimal VLMs Need Fewer Visual Tokens and More Parameters
  7. 2026From Patching to Pruning Visual Computation in Vision Language Models
  8. 2026Rethinking Token Reduction for Large Vision-Language Models