mlsanityData to 7 Oct 2026

Feather the Throttle: Revisiting Visual Token Pruning for Vision-Language Model Acceleration

arXiv 2412.13180 ↗First posted 17 Dec 2024cs.CVcited by 40

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

also in 1 more
Abstract (arXiv)

Recent works on accelerating Vision-Language Models achieve strong performance across a variety of vision-language tasks despite highly compressing visual information. In this work, we examine the popular acceleration approach of early pruning of visual tokens inside the language model. Surprisingly, we find that while strong performance is maintained across many tasks, it exhibits drastically different behavior for a subset of vision-centric tasks such as localization. Upon further investigation, we uncover a core issue with the acceleration approach where most tokens towards the top of the image are pruned away. Yet, on many benchmarks aiming to evaluate vision-centric capabilities, strong performance persists with the flawed pruning strategy, highlighting these benchmarks' limited ability to assess fine-grained visual capabilities. Based on these findings, we propose FEATHER (Fast and Effective Acceleration wiTH Ensemble cRiteria), a straightforward approach that resolves the discovered early-layer pruning issue and further enhances the preservation of relevant tokens via multistage pruning with early uniform sampling to ensure broad image coverage. With comparable computational savings, we find that FEATHER achieves more than 5x performance improvement on the vision-centric localization benchmarks compared to the original acceleration approach.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 36 of its 44 references that are in the map. First posted on arXiv 17 Dec 2024.

Builds on

36 of 44 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Models5 Dec
  2. 2024PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reduction22 Oct
  3. 2024ZipVL: Efficient Large Vision-Language Models with Dynamic Token Sparsification11 Oct
  4. 2024Recoverable Compression: A Multimodal Vision Token Recovery Mechanism Guided by Text Information2 Sep
  5. 2024HiRED: Attention-Guided Token Dropping for Efficient Inference of High-Resolution Vision-Language Models20 Aug
  6. 2024Efficient Large Multi-modal Models via Visual Context Compression28 Jun
  7. 2024LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inference26 Jun
  8. 2024Matryoshka Multimodal Models27 May
  9. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models22 Mar
  10. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models11 Mar

Other references · 26

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  8. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  9. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  10. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  11. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  12. 2019Towards VQA Models That Can Readcited by 1,696
14 more
  1. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  2. 2016Modeling Context in Referring Expressionscited by 1,054
  3. 2022Token Merging: Your ViT But Fastercited by 995
  4. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  5. 2016A Diagram Is Worth A Dozen Imagescited by 793
  6. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  7. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 336
  8. 2022Visual Spatial Reasoningcited by 288
  9. 2018TallyQA: Answering Complex Counting Questionscited by 152
  10. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
  11. 2024NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samplescited by 94
  12. 2023Revisiting Token Pruning for Object Detection and Instance Segmentationcited by 50
  13. 2021OCID-Ref: A 3D Robotic Dataset with Embodied Language for Clutter Scene Groundingcited by 15
  14. 2024On the token distance modeling ability of higher RoPE attention dimensioncited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 12; 2026: 28.

40 papers.

Sort:

By area:

  1. 2025Stop Looking for Important Tokens in Multimodal Language Models: Duplication Matters Morecited by 149
  2. 2025Don't Just Chase "Highlighted Tokens" in MLLMs: Revisiting Visual Holistic Context Retentioncited by 45
  3. 2025Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmarkcited by 37
  4. 2025FOCUS: Internal MLLM Representations for Efficient Fine-Grained Visual Question Answeringcited by 20
  5. 2025FlexSelect: Flexible Token Selection for Efficient Long Video Understandingcited by 17
  6. 2025Are We Using the Right Benchmark: An Evaluation Framework for Visual Token Compression Methodscited by 15
  7. 2026N"uwa: Mending the Spatial Integrity Torn by VLM Token Pruningcited by 13
  8. 2026The Model Knows Which Tokens Matter: Automatic Token Selection via Noise Gatingcited by 11
  9. 2025FiLA-Video: Spatio-Temporal Compression for Fine-Grained Long Video Understandingcited by 6
  10. 2025LVLM_CSP: Accelerating Large Vision Language Models via Clustering, Scattering, and Pruning for Reasoning Segmentationcited by 4
  11. 2025Attention Debiasing for Token Pruning in Vision Language Modelscited by 3
  12. 2026Keeping the Evidence Chain: Semantic Evidence Allocation for Training-Free Token Pruning in Video Temporal Groundingcited by 3
28 more
  1. 2026HAWK: Head Importance-Aware Visual Token Pruning in Multimodal Modelscited by 3
  2. 2025Streamline Without Sacrifice -- Squeeze out Computation Redundancy in LMMcited by 2
  3. 2025GreedyPrune: Retenting Critical Visual Token Set for Large Vision Language Modelscited by 2
  4. 2026SwiftVLM: Efficient Vision-Language Model Inference via Cross-Layer Token Bypasscited by 2
  5. 2026Reroute, Don't Remove: Recoverable Visual Token Routing for Vision-Language Modelscited by 2
  6. 2026CAPA: Contribution-Aware Pruning and FFN Approximation for Efficient Large Vision-Language Modelscited by 1
  7. 2026ASAP: Attention-Shift-Aware Pruning for Efficient LVLM Inferencecited by 1
  8. 2026SpecEyes: Accelerating Agentic Multimodal LLMs via Speculative Perception and Planningcited by 1
  9. 2026KiToke: Kernel-based Interval-aware Token Compression for Video Large Language Modelscited by 1
  10. 2026PARCEL: Pool-Anchored Resampling with Conditioned Elastic Queries for Efficient Vision-Language Understandingcited by 1
  11. 2026DiffPrune: differentiable information throttling for token pruning in vision-language modelscited by 1
  12. 2026When and Where to Look: Adaptive Visual Evidence Scheduling for Efficient Long Video Understandingcited by 1
  13. 2026Baseline-Relative Counterfactual Refinement for Bit-Aware Visual Token Communicationcited by 1
  14. 2026S2Prune: Spatially Structured Visual Token Pruning for Multimodal Large Language Modelscited by 1
  15. 2025ShaRP: SHAllow-LayeR Pruning for Efficient Video Large Language Modelscited by 0
  16. 2026LinMU: Multimodal Understanding Made Linearcited by 0
  17. 2026ConsensusDrop: Fusing Visual and Cross-Modal Saliency for Efficient Vision Language Modelscited by 0
  18. 2026NeuroCanvas: VLLM-Powered Robust Seizure Detection by Reformulating Multichannel EEG as Imagecited by 0
  19. 2026From Inheritance to Saturation: Disentangling the Evolution of Visual Redundancy for Architecture-Aware MLLM Inference Accelerationcited by 0
  20. 2026DCP-Prune: Ultra-Low Token Pruning with Distribution Consistency Preservationcited by 0
  21. 2026SepPrune:A Separator-based Pruning Framework for Efficient Multimodal Large Language Modelscited by 0
  22. 2026From Saliency to Discriminability: Rank-Preserving Visual Token Pruning for VLM Rerankerscited by 0
  23. 2026TRACE: Trajectory-robust Admission with Evidence Ordering for Efficient GUI Agentscited by 0
  24. 2026Pixel Decodability Is Not a Compression Signal: Causally Evaluating Importance Proxies for Visual KV-Cache Evictioncited by 0
  25. 2026SCOPD: Sparse-Context On-Policy Self-Distillation for Efficient Vision-Language Modelscited by 0
  26. 2026P4Q: Co-designing Token Pruning and Quantization for Vision-Language Model Accelerationcited by 0
  27. 2026Still There, No Longer Seen: Exposing Compression-Induced Risk in Large Vision-Language Modelscited by 0
  28. 2026DIPrune: Task-Aware Token Pruning with Dual Importance for Efficient Multimodal Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimization
  2. 2025ZSPAPrune: Zero-Shot Prompt-Aware Token Pruning for Vision-Language Models
  3. 2025A Glimpse to Compress: Dynamic Visual Token Pruning for Large Vision-Language Models
  4. 2026SlimVLM: Sensitivity-aware Dynamic Structured Pruning with Adaptive Visual Token Selection for Efficient Vision-Language Models
  5. 2026Beyond One-Size-Fits-All: Sample-Adaptive Strategy Routing for Vision Token Pruning in MLLMs
  6. 2026LearnPruner: Rethinking Attention-based Token Pruning in Vision Language Models
  7. 2026AgilePruner: An Empirical Study of Attention and Diversity for Adaptive Visual Token Pruning in Large Vision-Language Models
  8. 2026When Attention Collapses: Stage-Aware Visual Token Pruning from Structure to Semantics