mlsanityData to 14 Oct 2026

MM-ShiftKV: Decode-Aware Prefill-Stage KV Selection for Multimodal Large Language Models

arXiv 2607.22586 ↗First posted 9 Jun 2026cs.AI cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

Key-Value (KV) caching is essential for efficient inference in multimodal large language models (MLLMs), yet its memory footprint grows linearly with context length and becomes a major bottleneck due to the large number of visual tokens. Recent prefill-stage KV selection methods estimate KV importance from prefilling statistics, implicitly assuming that prefilling-time queries are representative of those encountered during decoding. We show that this assumption breaks down in multimodal inference, where decoding-time queries exhibit substantially larger variance than prefilling-stage representations, leading to unstable KV importance estimation under tight cache budgets. As a result, small ranking errors can disproportionately discard semantically critical visual tokens and degrade grounding and reasoning performance. We propose MM-ShiftKV, a training-free, decode-aware and strictly prefill-only KV selection method. MM-ShiftKV approximates decoding-time query behavior during prefilling by constructing variance-expanded query proxies and estimates prompt KV importance based on their aggregated attention mass. Experiments on multimodal benchmarks demonstrate that MM-ShiftKV consistently outperforms existing methods under strict KV-cache budgets. Our code is available at https://github.com/zjuDBxAI/MM-ShiftKV.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 25 of its 43 references that are in the map. First posted on arXiv 9 Jun 2026.

Builds on

25 of 43 references.

Closest ancestors · 4

Same direction, newest first

  1. 2026Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospects7 Apr
  2. 2025SparseMM: Head Sparsity Emerges from Visual Concept Responses in MLLMs5 Jun
  3. 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference6 Oct
  4. 2024LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inference26 Jun

Other references · 21

  1. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  2. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  3. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  4. 2019Towards VQA Models That Can Readcited by 1,718
  5. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  6. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  7. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  8. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  9. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  10. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  11. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  12. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 400
9 more
  1. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  2. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 307
  3. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 209
  4. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 197
  5. 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distributioncited by 74
  6. 2025RotateKV: Accurate and Robust 2-Bit KV Cache Quantization for LLMs via Outlier-Aware Adaptive Rotationscited by 46
  7. 2025SpecEE: Accelerating Large Language Model Inference with Speculative Early Exitingcited by 23
  8. 2025FloE: On-the-Fly MoE Inference on Memory-constrained GPUcited by 9
  9. 2026See the Forest for the Trees: Loosely Speculative Decoding via Visual-Semantic Guidance for Efficient Inference of Video LLMscited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Reducing Peak Memory Usage for Modern Multimodal Large Language Model Pipelines
  2. 2026Last But Not Least: Boundary Attention CalibratiON for Multimodal KV Cache Compression
  3. 2024Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference
  4. 2025MEDA: Dynamic KV Cache Allocation for Efficient Multimodal Long-Context Inference
  5. 2026HybridKV: Hybrid KV Cache Compression for Efficient Multimodal Large Language Model Inference
  6. 2026Stepwise Token Selection for Efficient Multimodal Large Language Models
  7. 2026Reformulating KV Cache Eviction Problem for Long-Context LLM Inference
  8. 2025Draft-based Approximate Inference for LLMs