mlsanityData to 7 Oct 2026

DRIP: Dynamic patch Reduction via Interpretable Pooling

arXiv 2510.25067 ↗First posted 29 Oct 2025cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

Recently, the advances in vision-language models, including contrastive pretraining and instruction tuning, have greatly pushed the frontier of multimodal AI. However, owing to the large-scale and hence expensive pretraining, the efficiency concern has discouraged researchers from attempting to pretrain a vision language model from scratch. In this work, we propose Dynamic patch Reduction via Interpretable Pooling (DRIP), which adapts to the input images and dynamically merges tokens in the deeper layers of a visual encoder. Our results on both ImageNet training from scratch and CLIP contrastive pretraining demonstrate a significant GFLOP reduction while maintaining comparable classification/zero-shot performance. To further validate our proposed method, we conduct continual pretraining on a large biology dataset, extending its impact into scientific domains.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 15 of its 25 references that are in the map. First posted on arXiv 29 Oct 2025.

Builds on

15 of 25 references.

References · 15

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  5. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  6. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  7. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  8. 2018Object Hallucination in Image Captioningcited by 661
  9. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  10. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  11. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 359
  12. 2021Hierarchical Transformers Are More Efficient Language Modelscited by 99
3 more
  1. 2022Efficient Transformers with Dynamic Token Poolingcited by 75
  2. 2024On the Vulnerability of LLM/VLM-Controlled Roboticscited by 15
  3. 2025Token Transforming: A Unified and Training-Free Token Compression Framework for Vision Transformer Accelerationcited by 9

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Looking to Learn: Token-wise Dynamic Gating for Low-Resource Vision-Language Modelling
  2. 2026From Patching to Pruning Visual Computation in Vision Language Models
  3. 2026Sparse CLIP: Co-Optimizing Interpretability and Performance in Contrastive Learning
  4. 2026FlashGaze: Training-Free Multi-Scale Patch Pruning For Efficient Video Understanding
  5. 2023Effective End-to-End Vision Language Pretraining with Semantic Visual Loss
  6. 2025Perception Encoder: The best visual embeddings are not at the output of the network
  7. 2026VIVAS: Vitalizing Visual Perception in VLM Pre-training via Vision-language Unified Autoregressive Supervision
  8. 2025Input-Adaptive Visual Preprocessing for Efficient Fast Vision-Language Model Inference