mlsanityData to 7 Oct 2026

ResKV: Reconstructing Omitted Attention Contributions for Fixed-Budget KV Cache Compression

arXiv 2607.29591 ↗First posted 31 Jul 2026cs.CLcited by 3

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

KV cache compression is essential for efficient long-context inference. Existing eviction methods permanently discard unselected tokens and consequently remove their aggregate contribution to attention. Merging-based alternatives preserve more information but can perturb retained keys and values that should remain exact. We observe that the information omitted by cache eviction can be formulated as residual statistics in both the numerator and denominator of softmax attention. Based on this observation, we propose ResKV, which divides a fixed KV budget into an exact main cache and a compact residual cache that reconstructs the contribution of omitted tokens. ResKV lets main-cache tokens and residual entries participate in the same softmax normalization, so residual entries restore both attention numerator and denominator mass rather than acting as a post-hoc correction. A construction-time validation proxy determines residual allocation for each layer and KV head, while a decode-time dynamic gate adjusts residual contributions for individual queries. Comprehensive evaluations on LongBench and RULER, covering query-aware and query-agnostic settings, multiple backbones, cache budgets, and representative compression baselines, demonstrate broad improvements under the same retained KV budget while preserving the practical efficiency of compressed decoding, including peak memory usage and long-context decode throughput.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 21 of its 27 references that are in the map. First posted on arXiv 31 Jul 2026.

Builds on

21 of 27 references.

Closest ancestors · 12

Same direction, newest first

  1. 2026SemantiCache: Efficient KV Cache Compression via Semantic Chunking and Clustered Merging15 Mar
  2. 2025OBCache: Optimal Brain KV Cache Pruning for Efficient Long-Context LLM Inference9 Oct
  3. 2024ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression4 Dec
  4. 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Heads22 Jul
  5. 2024Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Tasks11 Jul
  6. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  7. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  8. 2024SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget7 Apr
  9. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference14 Mar
  10. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  11. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun
  12. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 9

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  4. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  5. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  6. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  7. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  8. 2024Transformers are Multi-State RNNscited by 120
  9. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 114

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026CompKV: Compensation-Aware KV Selection for Long-Context LLM Inferencecited by 1
  2. 2026Counterexamples to Local Reconstruction Gain as a Proxy for Final Fidelity in Residual Completioncited by 0
  3. 2026CORE: COverage CAlibration and Evicted-Mass REdistribution for KV Cachecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026ResidualKV: Residual-Based KV Cache Compression for Efficient Long-Context Inference
  2. 2026KV2: A Self-Refining KV Cache
  3. 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distribution
  4. 2026Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction
  5. 2026Behavior-Preserving KV Cache Compression
  6. 2025SmallKV: Small Model Assisted Compensation of KV Cache Compression for Efficient LLM Inference
  7. 2026Last But Not Least: Boundary Attention CalibratiON for Multimodal KV Cache Compression
  8. 2026PuzzleKV: Page-Wise Low-Rank Decomposition for KV Cache Compression