mlsanityData to 7 Oct 2026

VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Acceleration

arXiv 2410.23317 ↗First posted 29 Oct 2024cs.CV cs.AI cs.CLcited by 41

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

also in 1 more
Abstract (arXiv)

Vision-Language Models (VLMs) have demonstrated impressive performance across a versatile set of tasks. A key challenge in accelerating VLMs is storing and accessing the large Key-Value (KV) cache that encodes long visual contexts, such as images or videos. While existing KV cache compression methods are effective for Large Language Models (LLMs), directly migrating them to VLMs yields suboptimal accuracy and speedup. To bridge the gap, we propose VL-Cache, a novel KV cache compression recipe tailored for accelerating VLM inference. In this paper, we first investigate the unique sparsity pattern of VLM attention by distinguishing visual and text tokens in prefill and decoding phases. Based on these observations, we introduce a layer-adaptive sparsity-aware cache budget allocation method that effectively distributes the limited cache budget across different layers, further reducing KV cache size without compromising accuracy. Additionally, we develop a modality-aware token scoring policy to better evaluate the token importance. Empirical results on multiple benchmark datasets demonstrate that retaining only 10% of KV cache achieves accuracy comparable to that with full cache. In a speed benchmark, our method accelerates end-to-end latency of generating 100 tokens by up to 2.33x and speeds up decoding by up to 7.08x, while reducing the memory footprint of KV cache in GPU by 90%.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 33 of its 40 references that are in the map. First posted on arXiv 29 Oct 2024.

Builds on

33 of 40 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024HiRED: Attention-Guided Token Dropping for Efficient Inference of High-Resolution Vision-Language Models20 Aug
  2. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models11 Mar

Other references · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2023Mistral 7Bcited by 3,065
  7. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  8. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  9. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  10. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  11. 2019Towards VQA Models That Can Readcited by 1,696
  12. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
19 more
  1. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  2. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  3. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  4. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  5. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  6. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  7. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  8. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  9. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 487
  10. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  11. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  12. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 420
  13. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 242
  14. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inferencecited by 148
  15. 2023PaLI-3 Vision Language Models: Smaller, Faster, Strongercited by 125
  16. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inferencecited by 124
  17. 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantizationcited by 99
  18. 2024ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identificationcited by 86
  19. 2024Effectively Compress KV Heads for LLMcited by 30

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 16; 2026: 23.

41 papers.

Sort:

By area:

  1. 2024FrameFusion: Combining Similarity and Importance for Video Token Reduction on Large Vision Language Modelscited by 52
  2. 2025AdaReTaKe: Adaptive Redundancy Reduction to Perceive Longer for Video-language Understandingcited by 45
  3. 2025Don't Just Chase "Highlighted Tokens" in MLLMs: Revisiting Visual Holistic Context Retentioncited by 45
  4. 2024ReTaKe: Reducing Temporal and Knowledge Redundancy for Long Video Understandingcited by 33
  5. 2025MMInference: Accelerating Pre-filling for Long-Context VLMs via Modality-Aware Permutation Sparse Attentioncited by 26
  6. 2025SpecVLM: Enhancing Speculative Decoding of Video LLMs via Verifier-Guided Token Pruningcited by 19
  7. 2025GUI-KV: Efficient GUI Agents via KV Cache with Spatio-Temporal Awarenesscited by 11
  8. 2025VisionSelector: End-to-End Learnable Visual Token Compression for Efficient Multimodal LLMscited by 11
  9. 2026Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Evictioncited by 10
  10. 2025EVICPRESS: Joint KV-Cache Compression and Eviction for Efficient LLM Servingcited by 9
  11. 2026Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospectscited by 8
  12. 2025Beyond Intermediate States: Explaining Visual Redundancy through Languagecited by 7
29 more
  1. 2025AirCache: Activating Inter-modal Relevancy KV Cache Compression for Efficient Large Vision-Language Model Inferencecited by 4
  2. 2025METok: Multi-Stage Event-based Token Compression for Efficient Long Video Understandingcited by 4
  3. 2025HiViS: Hiding Visual Tokens from the Drafter for Speculative Decoding in Vision-Language Modelscited by 4
  4. 2025V-Rex: Real-Time Streaming Video LLM Acceleration via Dynamic KV Cache Retrievalcited by 4
  5. 2026What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agentscited by 4
  6. 2025CalibQuant: 1-Bit KV Cache Quantization for Multimodal LLMscited by 3
  7. 2025EffiVLM-BENCH: A Comprehensive Benchmark for Evaluating Training-Free Acceleration in Large Vision-Language Modelscited by 3
  8. 2026Prefill-Time Intervention for Mitigating Hallucination in Large Vision-Language Modelscited by 3
  9. 2026TriAxialKV: Toward Extreme Low-Precision KV-Cache Quantization for Agentic Inference Taskscited by 3
  10. 2026A Policy-Driven Runtime Layer for Agentic LLM Servingcited by 3
  11. 2026Reflex: Real-Time VLA Control through Streaming Inferencecited by 3
  12. 2025Blink: Dynamic Visual Token Resolution for Enhanced Multimodal Understandingcited by 2
  13. 2026ST-Lite: Training-Free KV Cache Compression with Spatio-Trajectory Guidance for Long-Horizon GUI Agentscited by 2
  14. 2026AURA: Action-Gated Memory for Robot Policies at Constant VRAMcited by 2
  15. 2026Visual Access Boundaries in Vision-Language Model Reasoningcited by 2
  16. 2025Slim attention: cut your context memory in half without loss -- K-cache is all you need for MHAcited by 1
  17. 2026STaR-KV: Spatio-Temporal Adaptive Re-weighting for KV Cache Compression in GUI Vision-Language Modelscited by 1
  18. 2026HACK++: Towards More Effective Head-Aware Key-Value Compression for Efficient Visual Autoregressive Modelingcited by 1
  19. 2026Seen, Said, or Forgotten? A Causal Audit of Visual KV Memory Across Dialog Turnscited by 1
  20. 2026Learning Agent Execution for KV-Cache Management in Agentic Servingcited by 1
  21. 2026Q Cache: Visual Attention is Valuable in Less than Half of Decode Layers for Multimodal Large Language Modelcited by 0
  22. 2026Beyond Token Eviction: Mixed-Dimension Budget Allocation for Efficient KV Cache Compressioncited by 0
  23. 2026KVCapsule: Efficient Sequential KV Cache Compression for Vision-Language Models with Asymmetric Redundancycited by 0
  24. 2026Rotation-Aligned Key Channel Pruning for Efficient Vision-Language Model Inferencecited by 0
  25. 2026CIVIC: End-to-End Sequence Compactness for Efficient Vision-Language Modelscited by 0
  26. 2026AsymVLM: Asymmetric Token Pruning for Efficient Vision-Language Model Inferencecited by 0
  27. 2026FOVEA: Focused On-Demand Visual Evidence Adaptation for Cache-Friendly Multimodal Speculative Decodingcited by 0
  28. 2026Pixel Decodability Is Not a Compression Signal: Causally Evaluating Importance Proxies for Visual KV-Cache Evictioncited by 0
  29. 2026UniCache: Task- and Type-Aware KV Cache Compression for Unified Multimodal Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026VisCache: Visual KV Cache Pruning for Efficient Vision Large Language Model Inference
  2. 2026Make Your LVLM KV Cache More Lightweight
  3. 2025KV-Efficient VLA: A Method to Speed up Vision Language Models with RNN-Gated Chunked KV Cache
  4. 2025LightVLM: Acceleraing Large Multimodal Models with Pyramid Token Merging and KV Cache Compression
  5. 2026Hierarchical Compression of Vision-Language Model Benchmarks
  6. 2025Mixing Importance with Diversity: Joint Optimization for KV Cache Compression in Large Vision-Language Models
  7. 2024ZipVL: Efficient Large Vision-Language Models with Dynamic Token Sparsification
  8. 2026TGV-KV: Text-Grounded KV Eviction for Vision-Language Models