mlsanityData to 7 Oct 2026

SelKV: Selective KV Cache Merging with Per-Token Merge-or-Drop and Attention Compensation

arXiv 2607.16213 ↗First posted 19 May 2026cs.AIcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Large Language Models (LLMs) generate text autoregressively, relying on a key-value (KV) cache whose memory footprint grows linearly with context length, creating a major bottleneck. Recent compression methods mitigate this cost via token merging; however, these approaches often rely on indiscriminate aggregation, which degrades representations and introduces attention sag, a mismatch where merged tokens receive the same softmax mass as individual tokens despite encoding multiple inputs. We propose a training-free, dual-component framework for KV cache compression that addresses these limitations. First, a soft cosine gate adaptively modulates merging decisions based on value-vector similarity, suppressing or discarding dissimilar tokens to preserve semantic fidelity. Second, we introduce an attention-ratio compensation mechanism that applies a decoding-time logit bias derived from prefill attention statistics, correcting the softmax imbalance induced by merging. Evaluated on LongBench (16 English datasets) while retaining only 25% of the KV cache, our framework achieves strong compressed performance against representative one-shot baselines. It is especially robust on the evaluated grouped-query attention (GQA) models, maintaining nearlossless generation quality. Furthermore, the method outperforms the full-cache baseline on complex multi-document QA tasks and delivers a 3.3x decoding speedup at 100k tokens.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 16 of its 18 references that are in the map. First posted on arXiv 19 May 2026.

Builds on

16 of 18 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025LAVa: Layer-wise KV Cache Eviction with Dynamic Budget Allocation11 Sep
  2. 2025EvolKV: Evolutionary KV Cache Compression for LLM Inference10 Sep
  3. 2025WeightedKV: Attention Scores Weighted Key-Value Cache Merging for Large Language Models3 Mar
  4. 2025ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference1 Feb
  5. 2024EMS: Adaptive Evict-then-Merge Strategy for Head-wise KV Cache Compression Based on Global-Local Importance11 Dec
  6. 2024Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Tasks11 Jul
  7. 2024LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inference26 Jun
  8. 2024D2O: Dynamic Discriminative Operations for Efficient Long-Context Inference of Large Language Models18 Jun
  9. 2024Attention Score is not All You Need for Token Importance Indicator in KV Cache Reduction: Value Also Matters18 Jun
  10. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  11. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr

Other references · 5

  1. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  2. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  3. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  4. 2024ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identificationcited by 86
  5. 2023DiffRate : Differentiable Compression Rate for Efficient Vision Transformerscited by 78

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026GRKV: Global Regression for Training-Free KV Cache Compression in Long-Context LLMs
  2. 2025KeepKV: Achieving Periodic Lossless KV Cache Compression for Efficient LLM Inference
  3. 2025KVCompose: Efficient Structured KV Cache Compression with Composite Tokens
  4. 2024Compressing KV Cache for Long-Context LLM Inference with Inter-Layer Attention Similarity
  5. 2024More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compression
  6. 2026PuzzleKV: Page-Wise Low-Rank Decomposition for KV Cache Compression
  7. 2026A JoLT for the KV cache: Near-Lossless KV Cache Compression via Joint Rank-bit Allocation
  8. 2025ZSMerge: Zero-Shot KV Cache Compression for Memory-Efficient Long-Context LLMs