mlsanityData to 14 Oct 2026

How Query Visibility Changes KV-Cache Compression Rankings: A Matched-Budget Audit

arXiv 2607.11942 ↗First posted 11 Jul 2026cs.LG cs.AIcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

KV-cache compression methods are predominantly evaluated with the query appended to the context before compression -- a query-aware protocol. Yet the economic case for a compressed KV cache is reuse: compress a document once, answer many future questions against it. In that deployment, compression must happen query-agnostic -- before any question is seen. We present a matched-budget audit of six published compression methods against three trivial baselines on three open 7-9B models (144,300 paired evaluations on RULER-8192; 40,800 on LongBench; 50,000-resample paired bootstrap throughout). Everything is held fixed -- model, compression ratio, instances, decoding -- except the scoring rule. Three findings. (1) Query visibility changes the rankings: under the agnostic protocol, of the five audited methods that share a common attention backend, only KeyDiff beats a best-of-3 trivial baseline consistently (31 of 36 cells), and the most widely deployed method, SnapKV, loses to "keep the start and the recent window" on average (-0.066). (2) The per-method drop between the two protocols is ordered consistently with how visible the question is to each method's scoring signal, legible in its source code: from Delta=+0.198 for SnapKV (the question sits inside its 64-token observation window) down to Delta=+0.011 for KeyDiff (its score contains no query term at all).

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 13 of its 14 references that are in the map. First posted on arXiv 11 Jul 2026.

Builds on

13 of 14 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distribution1 Oct
  2. 2025Compactor: Calibrated Query-Agnostic KV Cache Compression with Approximate Leverage Scores10 Jul
  3. 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction29 May
  4. 2025KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environments21 Apr
  5. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methods13 Dec
  6. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
  7. 2024A Simple and Effective L2 Norm-Based Strategy for KV Cache Compression17 Jun
  8. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  9. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 4

  1. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  2. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  3. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  4. 2024Transformers are Multi-State RNNscited by 120

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026KV2: A Self-Refining KV Cache
  2. 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Heads
  3. 2026Cartridges++: KV Cache Compression without Off-Context Derailment
  4. 2026Faithfulness Is Not Free: Auditing Offline KV-Cache Quantization in Retrieval-Augmented Generation
  5. 2026CORE: COverage CAlibration and Evicted-Mass REdistribution for KV Cache
  6. 2024KV-Compress: Paged KV-Cache Compression with Variable Compression Rates per Attention Head
  7. 2026ResKV: Reconstructing Omitted Attention Contributions for Fixed-Budget KV Cache Compression
  8. 2025Rethinking Key-Value Cache Compression Techniques for Large Language Model Serving