mlsanityData to 7 Oct 2026

An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models

arXiv 2403.06764 ↗First posted 11 Mar 2024cs.CV cs.AI cs.CLcited by 629

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

also in 1 more
Abstract (arXiv)

In this study, we identify the inefficient attention phenomena in Large Vision-Language Models (LVLMs), notably within prominent models like LLaVA-1.5, QwenVL-Chat and Video-LLaVA. We find out that the attention computation over visual tokens is of extreme inefficiency in the deep layers of popular LVLMs, suggesting a need for a sparser approach compared to textual data handling. To this end, we introduce FastV, a versatile plug-and-play method designed to optimize computational efficiency by learning adaptive attention patterns in early layers and pruning visual tokens in subsequent ones. Our evaluations demonstrate FastV's ability to dramatically reduce computational costs (e.g., a 45 reduction in FLOPs for LLaVA-1.5-13B) without sacrificing performance in a wide range of image and video understanding tasks. The computational efficiency and performance trade-off of FastV are highly customizable and pareto-efficient. It can compress the FLOPs of a 13B-parameter model to achieve a lower budget than that of a 7B-parameter model, while still maintaining superior performance. We believe FastV has practical values for deployment of LVLMs in edge devices and commercial models. Code is released at https://github.com/pkunlp-icler/FastV.

Key facts. Cited by 629 papers in the map (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 34 of its 46 references that are in the map. First posted on arXiv 11 Mar 2024.

Builds on

34 of 46 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  8. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  9. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  10. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  11. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  12. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
22 more
  1. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  2. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  3. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  4. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  5. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  6. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  7. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  8. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  9. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  10. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  11. 2018nocaps: novel object captioning at scalecited by 444
  12. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  13. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 359
  14. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  15. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  16. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  17. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 235
  18. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 171
  19. 2023Silkie: Preference Distillation for Large Visual Language Modelscited by 128
  20. 2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuningcited by 126
  21. 2023Towards End-to-End Embodied Decision Making via Multi-modal Large Language Model: Explorations with GPT4-Vision and Beyondcited by 50
  22. 2024PCA-Bench: Evaluating Multimodal Large Language Models in Perception-Cognition-Action Chaincited by 29

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 72; 2025: 235; 2026: 322.

629 papers.

Sort:

By area:

  1. 2024Long Context Transfer from Language to Visioncited by 502
  2. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 420
  3. 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inferencecited by 378
  4. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Modelscited by 298
  5. 2024PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reductioncited by 283
  6. 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 226
  7. 2024VideoChat-Flash: Hierarchical Compression for Long-Context Video Modelingcited by 212
  8. 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
  9. 2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 200
  10. 2025DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Modelscited by 177
  11. 2025LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Tokencited by 164
  12. 2024Beyond Text-Visual Attention: Exploiting Visual Cues for Effective Token Pruning in VLMscited by 153
38 more
  1. 2024Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inferencecited by 137
  2. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
  3. 2025See What You Are Told: Visual Attention Sink in Large Multimodal Modelscited by 127
  4. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Modelscited by 122
  5. 2024Efficient Multimodal Large Language Models: A Surveycited by 95
  6. 2024DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Modelscited by 93
  7. 2025Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMscited by 92
  8. 2024Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Modelscited by 90
  9. 2024ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Modelscited by 89
  10. 2025EfficientVLA: Training-Free Acceleration and Compression for Vision-Language-Action Modelscited by 89
  11. 2025PD-VLA: Accelerating Vision-Language-Action Model Integrated with Action Chunking via Parallel Decodingcited by 88
  12. 2024MLLM can see? Dynamic Correction Decoding for Hallucination Mitigationcited by 87
  13. 2025VLA-Cache: Efficient Vision-Language-Action Manipulation via Adaptive Token Cachingcited by 83
  14. 2024Matryoshka Multimodal Modelscited by 82
  15. 2024VoCo-LLaMA: Towards Vision Compression with Large Language Modelscited by 78
  16. 2024FastVLM: Efficient Vision Encoding for Vision Language Modelscited by 76
  17. 2025CogVLA: Cognition-Aligned Vision-Language-Action Model via Instruction-Driven Routing & Sparsificationcited by 70
  18. 2024What's in the Image? A Deep-Dive into the Vision of Vision Language Modelscited by 66
  19. 2024Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsificationcited by 59
  20. 2025HoliTom: Holistic Token Merging for Fast Video Large Language Modelscited by 58
  21. 2025TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Modelcited by 57
  22. 2025SP-VLA: A Joint Model Scheduling and Token Pruning Approach for VLA Model Accelerationcited by 56
  23. 2024VideoLLM-MoD: Efficient Video-Language Streaming with Mixture-of-Depths Vision Computationcited by 55
  24. 2024SoftVQ-VAE: Efficient 1-Dimensional Continuous Tokenizercited by 55
  25. 20254D-VLA: Spatiotemporal Vision-Language-Action Pretraining with Cross-Scene Calibrationcited by 55
  26. 2024Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMscited by 54
  27. 2025InfiniPot-V: Memory-Constrained KV Cache Compression for Streaming Video Understandingcited by 54
  28. 2025StreamMem: Query-Agnostic KV Cache Memory for Streaming Video Understandingcited by 53
  29. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  30. 2025LiveVLM: Efficient Online Video Understanding via Streaming-Oriented KV Cache and Retrievalcited by 50
  31. 2025PACT: Pruning and Clustering-Based Token Reduction for Faster Visual Language Modelscited by 49
  32. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 49
  33. 2025FastVID: Dynamic Density Pruning for Fast Video Large Language Modelscited by 47
  34. 2024Efficient Large Multi-modal Models via Visual Context Compressioncited by 46
  35. 2024MBQ: Modality-Balanced Quantization for Large Vision-Language Modelscited by 45
  36. 2025Don't Just Chase "Highlighted Tokens" in MLLMs: Revisiting Visual Holistic Context Retentioncited by 45
  37. 2024ZipVL: Efficient Large Vision-Language Models with Dynamic Token Sparsificationcited by 43
  38. 2024AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruningcited by 43

50 most cited of 629 All 629 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospects
  2. 2024A-VL: Adaptive Attention for Large Vision-Language Models
  3. 2026CAPA: Contribution-Aware Pruning and FFN Approximation for Efficient Large Vision-Language Models
  4. 2025Growing a Multi-head Twig via Distillation and Reinforcement Learning to Accelerate Large Vision-Language Models
  5. 2025HiPrune: Hierarchical Attention for Efficient Token Pruning in Vision-Language Models
  6. 2025Efficient Vision-Language Reasoning via Adaptive Token Pruning
  7. 2025RedVTP: Training-Free Acceleration of Diffusion Vision-Language Models Inference via Masked Token-Guided Visual Token Pruning
  8. 2026TReVS: Integrating Textual Relevance and Visual Saliency for Efficient Vision-Language Model Token Pruning