mlsanityData to 14 Oct 2026

KV2: A Self-Refining KV Cache

arXiv 2610.03198 ↗First posted 2 Oct 2026cs.AI cs.CLcited by 1

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

The memory footprint of the key-value (KV) cache constrains the practical use of long-context models, and it dominates cost when one prefilled context must later serve many different queries. In this reusable setting, query-agnostic compression trades cost against quality: lightweight estimators are cheap but less accurate, whereas full-context reconstruction scoring is more accurate yet reprocesses the entire prompt. We introduce KV2, a query-agnostic KV-cache compression method based on selective reconstruction. KV2 first uses a lightweight proxy scorer to identify informative in-context tokens, then reprocesses only this subset to compute final eviction scores. On RULER, Needle-in-a-Haystack, and LongBench, KV2's margin over baselines widens as the budget tightens: on RULER 16K at a 2% KV-cache budget it improves the average score over the next-best baseline by more than 40 percentage points, and on LongBench it attains the highest average across 2%-10% budgets at lower compression-stage runtime and peak memory than full-context reconstruction. Reusable KV-cache compression thus does not require reprocessing the full context. Our code is available at https://anonymous.4open.science/r/KVsquared-0B97.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 23 of its 34 references that are in the map. First posted on arXiv 2 Oct 2026.

Builds on

23 of 34 references.

Closest ancestors · 13

Same direction, newest first

  1. 2026KVzap: Fast, Adaptive, and Faithful KV Cache Pruning12 Jan
  2. 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distribution1 Oct
  3. 2025Judge Q: Trainable Queries for Optimized Information Retention in KV Cache Eviction13 Sep
  4. 2025Compactor: Calibrated Query-Agnostic KV Cache Compression with Approximate Leverage Scores10 Jul
  5. 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction29 May
  6. 2025Lookahead Q-Cache: Achieving More Consistent KV Cache Eviction via Pseudo Query24 May
  7. 2024ThinK: Thinner Key Cache by Query-Driven Pruning30 Jul
  8. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
  9. 2024A Simple and Effective L2 Norm-Based Strategy for KV Cache Compression17 Jun
  10. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  11. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention23 Mar
  12. 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization28 Feb
1 more
  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 10

  1. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  2. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  4. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  5. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  6. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  7. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  8. 2023CoLT5: Faster Long-Range Transformers with Conditional Computationcited by 84
  9. 2024Preble: Efficient Distributed Prompt Scheduling for LLM Servingcited by 70
  10. 2025Value-Guided KV Compression for LLMs via Approximated CUR Decompositioncited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Rehearse Everything, Remember Nothing: Attic-KV Rehearses What Will Be Readcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Heads
  2. 2026EchoPress: Query-Agnostic KV Cache Pruning via Virtual Context Reconstruction
  3. 2026TaskPress: Query-Agnostic KV Cache Compression via Task-Guided Pruning
  4. 2026ResKV: Reconstructing Omitted Attention Contributions for Fixed-Budget KV Cache Compression
  5. 2026CORE: COverage CAlibration and Evicted-Mass REdistribution for KV Cache
  6. 2026PatchKV: Weight-Space Compensation of KV Cache
  7. 2026KVFetch: Temporal Prefetching for the Missing Half of KV Cache Compression
  8. 2026RestoreKV: Recovering Full-Cache Behavior Under Aggressive Query-Agnostic KV Cache Eviction