mlsanityData to 7 Oct 2026

PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reduction

arXiv 2410.17247 ↗First posted 22 Oct 2024cs.CV cs.CLcited by 283

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

In large vision-language models (LVLMs), images serve as inputs that carry a wealth of information. As the idiom "A picture is worth a thousand words" implies, representing a single image in current LVLMs can require hundreds or even thousands of tokens. This results in significant computational costs, which grow quadratically as input image resolution increases, thereby severely impacting the efficiency of both training and inference. Previous approaches have attempted to reduce the number of image tokens either before or within the early layers of LVLMs. However, these strategies inevitably result in the loss of crucial image information, ultimately diminishing model performance. To address this challenge, we conduct an empirical study revealing that all visual tokens are necessary for LVLMs in the shallow layers, and token redundancy progressively increases in the deeper layers of the model. To this end, we propose PyramidDrop, a visual redundancy reduction strategy for LVLMs to boost their efficiency in both training and inference with neglectable performance loss. Specifically, we partition the LVLM into several stages and drop part of the image tokens at the end of each stage with a pre-defined ratio, creating pyramid-like visual tokens across model layers. The dropping is based on a lightweight similarity calculation with a negligible time overhead. Extensive experiments demonstrate that PyramidDrop can achieve a 40% training time and 55% inference FLOPs acceleration of LLaVA-NeXT with comparable performance. Besides, the PyramidDrop could also serve as a plug-and-play strategy for inference acceleration without training, with better performance and lower inference cost than counterparts. Code is available at https://github.com/Cooperx521/PyramidDrop.

Key facts. Cited by 283 papers in the map (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 58 of its 69 references that are in the map. First posted on arXiv 22 Oct 2024.

Builds on

58 of 69 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference6 Oct
  2. 2024HiRED: Attention-Guided Token Dropping for Efficient Inference of High-Resolution Vision-Language Models20 Aug
  3. 2024DeCo: Decoupling Token Compression from Semantic Abstraction in Multimodal Large Language Models31 May
  4. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models22 Mar
  5. 2024LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Images18 Mar
  6. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models11 Mar
  7. 2023PuMer: Pruning and Merging Tokens for Efficient Vision Language Models27 May
  8. 2023CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformers27 May

Other references · 50

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  8. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  9. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  10. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  11. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  12. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
38 more
  1. 2019Towards VQA Models That Can Readcited by 1,696
  2. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  3. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  4. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  5. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  6. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  7. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  8. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  9. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  10. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  11. 2022Token Merging: Your ViT But Fastercited by 995
  12. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  13. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  14. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  15. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  16. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  17. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  18. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 468
  19. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  20. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  21. 2021InfographicVQAcited by 430
  22. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  23. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  24. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 359
  25. 2024Long-CLIP: Unlocking the Long-Text Capability of CLIPcited by 298
  26. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 285
  27. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
  28. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  29. 2024World Model on Million-Length Video And Language With Blockwise RingAttentioncited by 196
  30. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  31. 2024MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizationscited by 174
  32. 2024MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMscited by 89
  33. 2023OtterHD: A High-Resolution Multi-modality Modelcited by 72
  34. 2024Prism: A Framework for Decoupling and Assessing the Capabilities of VLMscited by 32
  35. 2024RAR: Retrieving And Ranking Augmented MLLMs for Visual Recognitioncited by 24
  36. 2023Prune Spatio-temporal Tokens by Semantic-aware Temporal Accumulationcited by 23
  37. 2024Deciphering Cross-Modal Alignment in Large Vision-Language Models with Modality Integration Ratecited by 20
  38. 2024PYRA: Parallel Yielding Re-Activation for Training-Inference Efficient Task Adaptationcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 114; 2026: 158.

283 papers.

Sort:

By area:

  1. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Modelscited by 298
  2. 2024Beyond Text-Visual Attention: Exploiting Visual Cues for Effective Token Pruning in VLMscited by 153
  3. 2025Stop Looking for Important Tokens in Multimodal Language Models: Duplication Matters Morecited by 149
  4. 2025Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMscited by 92
  5. 2025VideoRoPE: What Makes for Good Video Rotary Position Embedding?cited by 63
  6. 2025SIM-CoT: Supervised Implicit Chain-of-Thoughtcited by 63
  7. 2025HoliTom: Holistic Token Merging for Fast Video Large Language Modelscited by 58
  8. 2025FastVID: Dynamic Density Pruning for Fast Video Large Language Modelscited by 47
  9. 2025AdaReTaKe: Adaptive Redundancy Reduction to Perceive Longer for Video-language Understandingcited by 45
  10. 2025Don't Just Chase "Highlighted Tokens" in MLLMs: Revisiting Visual Holistic Context Retentioncited by 45
  11. 2024AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruningcited by 43
  12. 2025VScan: Rethinking Visual Token Reduction for Efficient Large Vision-Language Modelscited by 42
38 more
  1. 2025VisionThink: Smart and Efficient Vision Language Model via Reinforcement Learningcited by 41
  2. 2025StreamingTOM: Streaming Token Compression for Efficient Video Understandingcited by 41
  3. 2024Feather the Throttle: Revisiting Visual Token Pruning for Vision-Language Model Accelerationcited by 40
  4. 2024Filter, Correlate, Compress: Training-Free Token Reduction for MLLM Accelerationcited by 38
  5. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  6. 2025CapRL: Stimulating Dense Image Caption Capabilities via Reinforcement Learningcited by 36
  7. 2025When Large Vision-Language Model Meets Large Remote Sensing Imagery: Coarse-to-Fine Text-Guided Token Pruningcited by 35
  8. 2025The Better You Learn, The Smarter You Prune: Towards Efficient Vision-language-action Models via Differentiable Token Pruningcited by 34
  9. 2025Accelerating Streaming Video Large Language Models via Hierarchical Token Compressioncited by 34
  10. 2024ReTaKe: Reducing Temporal and Knowledge Redundancy for Long Video Understandingcited by 33
  11. 2025Global Compression Commander: Plug-and-Play Inference Acceleration for High-Resolution Large Vision-Language Modelscited by 32
  12. 2025Similarity-Aware Token Pruning: Your VLM but Fastercited by 32
  13. 2025FOLDER: Accelerating Multi-modal Large Language Models with Enhanced Performancecited by 31
  14. 2025Multi-Granular Spatio-Temporal Token Merging for Training-Free Acceleration of Video LLMscited by 30
  15. 2025Shifting AI Efficiency From Model-Centric to Data-Centric Compressioncited by 29
  16. 2025FlowCut: Rethinking Redundancy via Information Flow for Efficient Vision-Language Modelscited by 29
  17. 2024Accelerating Multimodal Large Language Models by Searching Optimal Vision Token Reductioncited by 28
  18. 2025Spatial Understanding from Videos: Structured Prompts Meet Simulation Datacited by 28
  19. 2025Scaling Vision Pre-Training to 4K Resolutioncited by 26
  20. 2025Towards Storage-Efficient Visual Document Retrieval: An Empirical Study on Reducing Patch-Level Embeddingscited by 26
  21. 2025LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMscited by 25
  22. 2025Variation-aware Vision Token Dropping for Faster Large Vision-Language Modelscited by 25
  23. 2025MR. Video: "MapReduce" is the Principle for Long Video Understandingcited by 23
  24. 2025SCOPE: Saliency-Coverage Oriented Token Pruning for Efficient Multimodel LLMscited by 23
  25. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 22
  26. 2025VFlowOpt: A Token Pruning Framework for LMMs with Visual Information Flow-Guided Optimizationcited by 21
  27. 2025OmniZip: Audio-Guided Dynamic Token Compression for Fast Omnimodal Large Language Modelscited by 21
  28. 2025Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimizationcited by 20
  29. 2025FOCUS: Internal MLLM Representations for Efficient Fine-Grained Visual Question Answeringcited by 20
  30. 2025FastDriveVLA: Efficient End-to-End Driving via Plug-and-Play Reconstruction-based Token Pruningcited by 20
  31. 2025TokenCarve: Information-Preserving Visual Token Compression in Multimodal Large Language Modelscited by 19
  32. 2025SpecVLM: Enhancing Speculative Decoding of Video LLMs via Verifier-Guided Token Pruningcited by 19
  33. 2025AdaptVision: Efficient Vision-Language Models via Adaptive Visual Acquisitioncited by 19
  34. 2025Hybrid Token Compression for Vision-Language Modelscited by 19
  35. 2025A Glimpse to Compress: Dynamic Visual Token Pruning for Large Vision-Language Modelscited by 18
  36. 2025CATP: Contextually Adaptive Token Pruning for Efficient and Enhanced Multimodal In-Context Learningcited by 18
  37. 2025FlexSelect: Flexible Token Selection for Efficient Long Video Understandingcited by 17
  38. 2024p-MoD: Building Mixture-of-Depths MLLMs via Progressive Ratio Decaycited by 16

50 most cited of 283 All 283 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024ZipVL: Efficient Large Vision-Language Models with Dynamic Token Sparsification
  2. 2025LVPruning: An Effective yet Simple Language-Guided Vision Token Pruning Approach for Multi-modal Large Language Models
  3. 2025EffiVLM-BENCH: A Comprehensive Benchmark for Evaluating Training-Free Acceleration in Large Vision-Language Models
  4. 2025Short-LVLM: Compressing and Accelerating Large Vision-Language Models by Pruning Redundant Layers
  5. 2024Inference Optimal VLMs Need Fewer Visual Tokens and More Parameters
  6. 2025LightVLM: Acceleraing Large Multimodal Models with Pyramid Token Merging and KV Cache Compression
  7. 2024ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models
  8. 2026AsymVLM: Asymmetric Token Pruning for Efficient Vision-Language Model Inference