mlsanityData to 7 Oct 2026

LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models

arXiv 2403.15388 ↗First posted 22 Mar 2024cs.CV cs.AI cs.CLcited by 394

On the map: LLM reasoning → Multimodal reasoning

also in 2 more
Abstract (arXiv)

Large Multimodal Models (LMMs) have shown significant visual reasoning capabilities by connecting a visual encoder and a large language model. LMMs typically take in a fixed and large amount of visual tokens, such as the penultimate layer features in the CLIP visual encoder, as the prefix content. Recent LMMs incorporate more complex visual inputs, such as high-resolution images and videos, which further increases the number of visual tokens significantly. However, due to the inherent design of the Transformer architecture, the computational costs of these models tend to increase quadratically with the number of input tokens. To tackle this problem, we explore a token reduction mechanism that identifies significant spatial redundancy among visual tokens. In response, we propose PruMerge, a novel adaptive visual token reduction strategy that significantly reduces the number of visual tokens without compromising the performance of LMMs. Specifically, to metric the importance of each token, we exploit the sparsity observed in the visual encoder, characterized by the sparse distribution of attention scores between the class token and visual tokens. This sparsity enables us to dynamically select the most crucial visual tokens to retain. Subsequently, we cluster the selected (unpruned) tokens based on their key similarity and merge them with the unpruned tokens, effectively supplementing and enhancing their informational content. Empirically, when applied to LLaVA-1.5, our approach can compress the visual tokens by 14 times on average, and achieve comparable performance across diverse visual question-answering and reasoning tasks. Code and checkpoints are at https://llava-prumerge.github.io/.

Key facts. Cited by 394 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 49 of its 64 references that are in the map. First posted on arXiv 22 Mar 2024.

Builds on

49 of 64 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  2. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering20 Sep

Other references · 47

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2016Layer Normalizationcited by 5,579
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  11. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  12. 2023Qwen Technical Reportcited by 3,273
35 more
  1. 2023Mistral 7Bcited by 3,065
  2. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  3. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  4. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  5. 2019Towards VQA Models That Can Readcited by 1,696
  6. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  7. 2020Reformer: The Efficient Transformercited by 1,658
  8. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  9. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  10. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  11. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  12. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  13. 2022Token Merging: Your ViT But Fastercited by 995
  14. 2023A Survey on Multimodal Large Language Modelscited by 898
  15. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  16. 2020Efficient Transformers: A Surveycited by 784
  17. 2020A Survey on Visual Transformercited by 653
  18. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
  19. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 359
  20. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  21. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  22. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  23. 2021Adaptive Token Sampling For Efficient Vision Transformerscited by 181
  24. 2024MobileVLM V2: Faster and Stronger Baseline for Vision Language Modelcited by 172
  25. 2024TinyLLaVA: A Framework of Small-scale Large Multimodal Modelscited by 164
  26. 2023ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Promptscited by 160
  27. 2023ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Modelscited by 156
  28. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  29. 2024LLaVA-Phi: Efficient Multi-Modal Assistant with Small Language Modelcited by 128
  30. 2023PB-LLM: Partially Binarized Large Language Modelscited by 80
  31. 2023TinyGPT-V: Efficient Multimodal Large Language Model via Small Backbonescited by 77
  32. 2023Which Tokens to Use? Investigating Token Reduction in Vision Transformerscited by 59
  33. 2023CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformerscited by 44
  34. 2024Small Language Model Meets with Reinforced Vision Vocabularycited by 43
  35. 2022Adaptive Sparse ViT: Towards Learnable Adaptive Token Pruning by Fully Exploiting Self-Attentioncited by 35

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 65; 2025: 146; 2026: 183.

394 papers.

Sort:

By area:

  1. 2024Long Context Transfer from Language to Visioncited by 502
  2. 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inferencecited by 378
  3. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Modelscited by 298
  4. 2024PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reductioncited by 283
  5. 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 226
  6. 2025DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Modelscited by 177
  7. 2024TokenPacker: Efficient Visual Projector for Multimodal LLMcited by 170
  8. 2025LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Tokencited by 164
  9. 2024Beyond Text-Visual Attention: Exploiting Visual Cues for Effective Token Pruning in VLMscited by 153
  10. 2025Magma: A Foundation Model for Multimodal AI Agentscited by 138
  11. 2024Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inferencecited by 137
  12. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
38 more
  1. 2024Efficient Multimodal Large Language Models: A Surveycited by 95
  2. 2024LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inferencecited by 93
  3. 2024DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Modelscited by 93
  4. 2025Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMscited by 92
  5. 2024Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Modelscited by 90
  6. 2024ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Modelscited by 89
  7. 2025EfficientVLA: Training-Free Acceleration and Compression for Vision-Language-Action Modelscited by 89
  8. 2024Matryoshka Multimodal Modelscited by 82
  9. 2024FastVLM: Efficient Vision Encoding for Vision Language Modelscited by 76
  10. 2025CogVLA: Cognition-Aligned Vision-Language-Action Model via Instruction-Driven Routing & Sparsificationcited by 70
  11. 2024Multi-Stage Vision Token Dropping: Towards Efficient Multimodal Large Language Modelcited by 67
  12. 2024PruneVid: Visual Token Pruning for Efficient Video Large Language Modelscited by 64
  13. 2024Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsificationcited by 59
  14. 2025HoliTom: Holistic Token Merging for Fast Video Large Language Modelscited by 58
  15. 2025TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Modelcited by 57
  16. 2025SP-VLA: A Joint Model Scheduling and Token Pruning Approach for VLA Model Accelerationcited by 56
  17. 2024VideoLLM-MoD: Efficient Video-Language Streaming with Mixture-of-Depths Vision Computationcited by 55
  18. 2024FrameFusion: Combining Similarity and Importance for Video Token Reduction on Large Vision Language Modelscited by 52
  19. 2024Benchmarking Multi-Image Understanding in Vision and Language Models: Perception, Knowledge, Reasoning, and Multi-Hop Reasoningcited by 50
  20. 2025PACT: Pruning and Clustering-Based Token Reduction for Faster Visual Language Modelscited by 49
  21. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 49
  22. 2025FastVID: Dynamic Density Pruning for Fast Video Large Language Modelscited by 47
  23. 2024MBQ: Modality-Balanced Quantization for Large Vision-Language Modelscited by 45
  24. 2025Don't Just Chase "Highlighted Tokens" in MLLMs: Revisiting Visual Holistic Context Retentioncited by 45
  25. 2024AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruningcited by 43
  26. 2025VScan: Rethinking Visual Token Reduction for Efficient Large Vision-Language Modelscited by 42
  27. 2025StreamingTOM: Streaming Token Compression for Efficient Video Understandingcited by 41
  28. 2024Feather the Throttle: Revisiting Visual Token Pruning for Vision-Language Model Accelerationcited by 40
  29. 2024DocKylin: A Large Multimodal Model for Visual Document Understanding with Efficient Visual Slimmingcited by 35
  30. 2024A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMscited by 35
  31. 2025When Large Vision-Language Model Meets Large Remote Sensing Imagery: Coarse-to-Fine Text-Guided Token Pruningcited by 35
  32. 2024xGen-MM-Vid (BLIP-3-Video): You Only Need 32 Tokens to Represent a Video Even in VLMscited by 34
  33. 2025The Better You Learn, The Smarter You Prune: Towards Efficient Vision-language-action Models via Differentiable Token Pruningcited by 34
  34. 2024TextHawk2: A Large Vision-Language Model Excels in Bilingual OCR and Grounding with 16x Fewer Tokenscited by 32
  35. 2025Global Compression Commander: Plug-and-Play Inference Acceleration for High-Resolution Large Vision-Language Modelscited by 32
  36. 2025FOLDER: Accelerating Multi-modal Large Language Models with Enhanced Performancecited by 31
  37. 2025Multi-Granular Spatio-Temporal Token Merging for Training-Free Acceleration of Video LLMscited by 30
  38. 2025Shifting AI Efficiency From Model-Centric to Data-Centric Compressioncited by 29

50 most cited of 394 All 394 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025TrimTokenator-LC: Towards Adaptive Visual Token Pruning for Large Multimodal Models with Long Contexts
  2. 2025TrimTokenator: Towards Adaptive Visual Token Pruning for Large Multimodal Models
  3. 2026SinkPruner: Sink-Free Visual Token Pruning for Multimodal Large Language Models
  4. 2026One Layer's Trash is Another Layer's Treasure: Adaptive Layer-wise Visual Token Selection in LVLMs
  5. 2026How Many Visual Tokens Do Multimodal Language Models Need? Scaling Visual Token Pruning with F^3A
  6. 2026Efficient Token Pruning for LLaDA-V
  7. 2025LFTR: Learning-Free Token Reduction for Multimodal Large Language Models
  8. 2026STAR-Pro: Stage-Wise Token Adaptive Reduction with Progressive Refinement for Efficient Large Vision-Language Models