mlsanityData to 14 Oct 2026

PuMer: Pruning and Merging Tokens for Efficient Vision Language Models

arXiv 2305.17530 ↗First posted 27 May 2023cs.CV cs.AI cs.CLcited by 53

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

Large-scale vision language (VL) models use Transformers to perform cross-modal interactions between the input text and image. These cross-modal interactions are computationally expensive and memory-intensive due to the quadratic complexity of processing the input image and text. We present PuMer: a token reduction framework that uses text-informed Pruning and modality-aware Merging strategies to progressively reduce the tokens of input image and text, improving model inference speed and reducing memory footprint. PuMer learns to keep salient image tokens related to the input text and merges similar textual and visual tokens by adding lightweight token reducer modules at several cross-modal layers in the VL model. Training PuMer is mostly the same as finetuning the original VL model but faster. Our evaluation for two vision language models on four downstream VL tasks shows PuMer increases inference throughput by up to 2x and reduces memory footprint by over 50% while incurring less than a 1% accuracy drop.

Key facts. Cited by 53 papers in the map (data through 14 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 33 of its 45 references that are in the map. First posted on arXiv 27 May 2023.

Builds on

33 of 45 references.

References · 33

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  9. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  10. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  11. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  12. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
21 more
  1. 2022Token Merging: Your ViT But Fastercited by 1,021
  2. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 864
  3. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  4. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647
  5. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 531
  6. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  7. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 459
  8. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 362
  9. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 273
  10. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  11. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 256
  12. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 226
  13. 2021Block Pruning For Faster Transformerscited by 177
  14. 2021TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?cited by 127
  15. 2020The Right Tool for the Job: Matching Model and Instance Complexitiescited by 126
  16. 2021Compressing Visual-linguistic Model via Knowledge Distillationcited by 86
  17. 2022Efficient Transformers with Dynamic Token Poolingcited by 76
  18. 2021Token Pooling in Vision Transformerscited by 69
  19. 2020MiniVLM: A Smaller and Faster Vision-Language Modelcited by 51
  20. 2021A Unified Pruning Framework for Vision Transformerscited by 40
  21. 2018Visual Entailment Task for Visually-Grounded Language Learningcited by 36

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 12; 2025: 27; 2026: 13.

53 papers.

Sort:

By area:

  1. 2024PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reductioncited by 294
  2. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 128
  3. 2025FastVGGT: Training-Free Acceleration of Visual Geometry Transformercited by 98
  4. 2023MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Trainingcited by 96
  5. 2024Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Modelscited by 94
  6. 2025VLA-Cache: Efficient Vision-Language-Action Manipulation via Adaptive Token Cachingcited by 86
  7. 2025VL-JEPA: Joint Embedding Predictive Architecture for Vision-languagecited by 68
  8. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 50
  9. 2024SafeWatch: An Efficient Safety-Policy Following Video Guardrail Model with Transparent Explanationscited by 38
  10. 2025What Kind of Visual Tokens Do We Need? Training-free Visual Token Pruning for Multi-modal Large Language Models from the Perspective of Graphcited by 36
  11. 2024Accelerating Transformers with Spectrum-Preserving Token Mergingcited by 35
  12. 2025Shifting AI Efficiency From Model-Centric to Data-Centric Compressioncited by 29
38 more
  1. 2024ALGM: Adaptive Local-then-Global Token Merging for Efficient Semantic Segmentation with Plain Vision Transformerscited by 28
  2. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 25
  3. 2025MobileCLIP2: Improving Multi-Modal Reinforced Trainingcited by 21
  4. 2025Hybrid Token Compression for Vision-Language Modelscited by 20
  5. 2024Efficient Vision-Language Models by Summarizing Visual Tokens into Compact Registerscited by 19
  6. 2025LiteVGGT: Boosting Vanilla VGGT via Geometry-aware Cached Token Mergingcited by 17
  7. 2024Rethinking Token Reduction for State Space Modelscited by 15
  8. 2025Evict3R: Training-Free Token Eviction for Memory-Bounded Streaming Visual Geometry Transformerscited by 14
  9. 2025Accelerating Vision Transformers with Adaptive Patch Sizescited by 13
  10. 2025MergeVQ: A Unified Framework for Visual Generation and Representation with Disentangled Token Merging and Quantizationcited by 12
  11. 2026The Model Knows Which Tokens Matter: Automatic Token Selection via Noise Gatingcited by 11
  12. 2025Co-Me: Confidence-Guided Token Merging for Visual Geometric Transformerscited by 10
  13. 2024PaPr: Training-Free One-Step Patch Pruning with Lightweight ConvNets for Faster Inferencecited by 9
  14. 2025Grounding-Aware Token Pruning: Recovering from Drastic Performance Drops in Visual Grounding Caused by Pruningcited by 9
  15. 2026Attend Before Attention: Efficient and Scalable Video Understanding via Autoregressive Gazingcited by 9
  16. 2024Famba-V: Fast Vision Mamba with Cross-Layer Token Fusioncited by 8
  17. 2025A Survey on Mamba Architecture for Vision Applicationscited by 8
  18. 2025One Trajectory, One Token: Grounded Video Tokenization via Panoptic Sub-object Trajectorycited by 7
  19. 2024Deploying Foundation Model Powered Agent Services: A Surveycited by 4
  20. 2025Efficient Token Compression for Vision Transformer with Spatial Information Preservedcited by 4
  21. 2025RedVTP: Training-Free Acceleration of Diffusion Vision-Language Models Inference via Masked Token-Guided Visual Token Pruningcited by 4
  22. 2024Visual-Word Tokenizer: Beyond Fixed Sets of Tokens in Vision Transformerscited by 2
  23. 2025STAR: Stage-Wise Attention-Guided Token Reduction for Efficient Large Vision-Language Models Inferencecited by 2
  24. 2025QLIP: A Dynamic Quadtree Vision Prior Enhances MLLM Performance Without Retrainingcited by 2
  25. 2025Video, How Do Your Tokens Merge?cited by 2
  26. 2026Temporal Aware Pruning for Efficient Diffusion-based Video Generationcited by 2
  27. 2025NEURAL: Attention-Guided Pruning for Unified Multimodal Resource-Constrained Clinical Evaluationcited by 1
  28. 2025AttAnchor: Guiding Cross-Modal Token Alignment in VLMs with Attention Anchorscited by 1
  29. 2025AdaTok: Adaptive Token Compression with Object-Aware Representations for Efficient Multimodal LLMscited by 1
  30. 2026Learning to Accelerate Vision-Language-Action Models through Adaptive Visual Token Cachingcited by 1
  31. 2026ASAP: Attention-Shift-Aware Pruning for Efficient LVLM Inferencecited by 1
  32. 2026LookWhen? Fast Video Recognition by Learning When, Where, and What to Computecited by 1
  33. 2026OTT-Vid: Optimal Transport Temporal Token Compression for Video Large Language Modelscited by 1
  34. 2026Fast Enough to Act: Spatio-Temporal Visual Token Merging for Low-Latency Robotic VLMs and VLAscited by 1
  35. 2025Dynamic Pyramid Network for Efficient Multimodal Large Language Modelcited by 0
  36. 2025Improving MLLM Training Efficiency via Stage-Aware Sparsitycited by 0
  37. 2026Taking Shortcuts for Categorical VQA Using Super Neuronscited by 0
  38. 2026Towards Efficient Large Vision-Language Models: A Comprehensive Survey on Inference Strategiescited by 0

50 most cited of 53 All 53 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026SwiftVLM: Efficient Vision-Language Model Inference via Cross-Layer Token Bypass
  2. 2026From Patching to Pruning Visual Computation in Vision Language Models
  3. 2025LVPruning: An Effective yet Simple Language-Guided Vision Token Pruning Approach for Multi-modal Large Language Models
  4. 2025Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimization
  5. 2025Object-Centric Vision Token Pruning for Vision Language Models
  6. 2025Efficient Vision-Language Reasoning via Adaptive Token Pruning
  7. 2022EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruning
  8. 2025VScan: Rethinking Visual Token Reduction for Efficient Large Vision-Language Models