# KVCapsule: Efficient Sequential KV Cache Compression for Vision-Language Models with Asymmetric Redundancy

arXiv: https://arxiv.org/abs/2605.16439 · HTML: https://mlsanity.com/paper/2605.16439/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 32 of its 34 references that are in the map.
- First posted on arXiv 14 May 2026.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [Visual token pruning and efficiency](https://mlsanity.com/mllm/visual-token-pruning-and-efficiency/)
- Efficient LLM inference and serving / [Visual token pruning in multimodal models](https://mlsanity.com/inference/visual-token-pruning-in-multimodal-models/)

## Builds on (most cited first, 30 of 32)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://mlsanity.com/paper/2504.10479/) (2025)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024)
- [SnapKV: LLM Knows What You are Looking for Before Generation](https://mlsanity.com/paper/2404.14469/) (2024)
- [KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization](https://mlsanity.com/paper/2401.18079/) (2024)
- [HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models](https://mlsanity.com/paper/2310.14566/) (2023)
- [A Corpus for Reasoning About Natural Language Grounded in Photographs](https://mlsanity.com/paper/1811.00491/) (2018)
- [VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Models](https://mlsanity.com/paper/2407.11691/) (2024)
- [MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?](https://mlsanity.com/paper/2408.13257/) (2024)
- [MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding](https://mlsanity.com/paper/2406.14515/) (2024)
- [LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inference](https://mlsanity.com/paper/2406.18139/) (2024)
- [Eigen Attention: Attention in Low-Rank Space for KV Cache Compression](https://mlsanity.com/paper/2408.05646/) (2024)
- [QAQ: Quality Adaptive Quantization for LLM KV Cache](https://mlsanity.com/paper/2403.04643/) (2024)
- [Palu: Compressing KV-Cache with Low-Rank Projection](https://mlsanity.com/paper/2407.21118/) (2024)
- [TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Model](https://mlsanity.com/paper/2503.18278/) (2025)
- [LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategy](https://mlsanity.com/paper/2410.03111/) (2024)
- [VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Acceleration](https://mlsanity.com/paper/2410.23317/) (2024)
- [MEGA-Bench: Scaling Multimodal Evaluation to over 500 Real-World Tasks](https://mlsanity.com/paper/2410.10563/) (2024)
- [Cross-Self KV Cache Pruning for Efficient Vision-Language Inference](https://mlsanity.com/paper/2412.04652/) (2024)
- [MadaKV: Adaptive Modality-Perception KV Cache Eviction for Efficient Multimodal Long-Context Inference](https://mlsanity.com/paper/2506.15724/) (2025)
- [SALS: Sparse Attention in Latent Space for KV cache Compression](https://mlsanity.com/paper/2510.24273/) (2025)
- [AKVQ-VL: Attention-Aware KV Cache Adaptive 2-Bit Quantization for Vision-Language Models](https://mlsanity.com/paper/2501.15021/) (2025)
- [PureKV: Plug-and-Play KV Cache Optimization with Spatial-Temporal Sparse Attention for Vision-Language Large Models](https://mlsanity.com/paper/2510.25600/) (2025)

## Abstract (arXiv)

Vision-Language Models (VLMs) have emerged as a critical and fast-growing extension of Large Language Models (LLMs) that enable multimodal reasoning through both text and image inputs. Although VLMs enrich the capabilities of language models, they also inherit and amplify key computational bottlenecks: the memory overhead caused by the large key-value (KV) cache during autoregressive decoding. This challenge is particularly severe in VLMs, where images produce longer token sequences and denser feature representations compared to text. Moreover, the spatial and information-rich nature of vision tokens introduces structured attention patterns that make many LLM-oriented KV cache compression techniques ineffective when applied directly to VLMs. In this work, we conduct a detailed empirical analysis of the behavior of vision tokens, highlighting the critical differences from purely text-based models. Based on these insights, we propose KVCapsule, a novel KV cache compression framework for vision tokens. KVCapsule keeps the pretrained VLM backbone frozen, requires no modification to the attention computation modules, and can be integrated into existing VLMs through lightweight compression and reconstruction components. We evaluate KVCapsule on multiple VLMs and benchmark tasks, demonstrating up to 2x improvement in TPS and 2.4x reduction in KV cache memory at a 60% compression ratio, with negligible degradation in accuracy or response quality. Our findings offer practical pathways to scale VLM inference under constrained memory budgets and inspire further research into structure-aware cache compression for multimodal models.