mlsanityData to 14 Oct 2026

KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction

arXiv 2505.23416 ↗First posted 29 May 2025cs.DB cs.LGcited by 95

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Transformer-based large language models (LLMs) cache context as key-value (KV) pairs during inference. As context length grows, KV cache sizes expand, leading to substantial memory overhead and increased attention latency. This paper introduces KVzip, a query-agnostic KV cache eviction method enabling effective reuse of compressed KV caches across diverse queries. KVzip quantifies the importance of a KV pair using the underlying LLM to reconstruct original contexts from cached KV pairs, subsequently evicting pairs with lower importance. Extensive empirical evaluations demonstrate that KVzip reduces KV cache size by 3-4× and FlashAttention decoding latency by approximately 2×, with negligible performance loss in question-answering, retrieval, reasoning, and code comprehension tasks. Evaluations include various models such as LLaMA3.1, Qwen2.5, and Gemma3, with context lengths reaching up to 170K tokens. KVzip significantly outperforms existing query-aware KV eviction methods, which suffer from performance degradation even at a 90% cache budget ratio under multi-query scenarios.

Key facts. Cited by 95 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 51 of its 61 references that are in the map. First posted on arXiv 29 May 2025.

Builds on

51 of 61 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methods13 Dec
  2. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads14 Oct
  3. 2024NACL: A General and Effective KV Cache Eviction Framework for LLMs at Inference Time7 Aug
  4. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
  5. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  6. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  7. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  8. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference21 May
  9. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  10. 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization28 Feb
  11. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  12. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
2 more
  1. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun
  2. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 37

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  8. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  9. 2018Deep contextualized word representationscited by 2,290
  10. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  11. 2025Gemma 3 Technical Reportcited by 1,824
  12. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
25 more
  1. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  2. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  3. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  4. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  5. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  6. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  7. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  8. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 518
  9. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 472
  10. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  11. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
  12. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 260
  13. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 252
  14. 2016Professor Forcing: A New Algorithm for Training Recurrent Networkscited by 216
  15. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 208
  16. 2025Qwen2.5-1M Technical Reportcited by 164
  17. 2021Learned Token Pruning for Transformerscited by 139
  18. 2024Transformers are Multi-State RNNscited by 120
  19. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 116
  20. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 111
  21. 2023Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformerscited by 67
  22. 2023Compressed Context Memory For Online Language Model Interactioncited by 43
  23. 2024Don't Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Taskscited by 38
  24. 2024InfiniPot: Infinite Context Processing on Memory-Constrained LLMscited by 27
  25. 2025Beyond RAG: Task-Aware KV Cache Compression for Comprehensive Knowledge Reasoningcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 12; 2026: 83.

95 papers.

Sort:

By area:

  1. 2025StreamMem: Query-Agnostic KV Cache Memory for Streaming Video Understandingcited by 58
  2. 2025ThinKV: Thought-Adaptive KV Cache Compression for Efficient Reasoning Modelscited by 21
  3. 2026Parallelizing Tool Execution and LLM Generation for Low-Latency Agent Servingcited by 20
  4. 2026Fast KVzip: Efficient and Accurate LLM Inference with Gated KV Evictioncited by 16
  5. 2026KVzap: Fast, Adaptive, and Faithful KV Cache Pruningcited by 15
  6. 2026Learning to Evict from Key-Value Cachecited by 14
  7. 2025OBCache: Optimal Brain KV Cache Pruning for Efficient Long-Context LLM Inferencecited by 10
  8. 2025Which Heads Matter for Reasoning? RL-Guided KV Cache Compressioncited by 10
  9. 2025EVICPRESS: Joint KV-Cache Compression and Eviction for Efficient LLM Servingcited by 10
  10. 2026LookaheadKV: Fast and Accurate KV Cache Eviction by Glimpsing into the Future without Generationcited by 10
  11. 2026ResidualKV: Residual-Based KV Cache Compression for Efficient Long-Context Inferencecited by 9
  12. 2025EpiCache: Episodic KV Cache Management for Long-Term Conversation on Resource-Constrained Environmentscited by 7
38 more
  1. 2025G-KV: Decoding-Time KV Cache Eviction with Global Attentioncited by 7
  2. 2026Training Transformers for KV Cache Compressibilitycited by 7
  3. 2026Predicting Future Utility: Global Combinatorial Optimization for Task-Agnostic KV Cache Evictioncited by 6
  4. 2026Cartridges at Scale: Training Modular KV Caches over Large Document Collectionscited by 6
  5. 2026KVEraser: Learning to Steer KV Cache for Efficient Localized Context Erasingcited by 6
  6. 2026When Does Value-Aware KV Eviction Help? A Fixed-Contract Diagnostic for Non-Monotone Cache Compressioncited by 5
  7. 2026Towards Fast and Effective Long Video Understanding of Multimodal Large Language Models via Adaptive Quasi-Gaussian Samplingcited by 5
  8. 2025KV Admission: Learning What to Write for Efficient Long-Context LLM Inferencecited by 4
  9. 2026Stochastic KV Routing: Enabling Adaptive Depth-Wise Cache Sharingcited by 4
  10. 2026A Simple Plug-in for Improving Eviction-Based KV Cache Compressioncited by 4
  11. 2026Still: Amortized KV Cache Compaction in a Single Forward Passcited by 4
  12. 2026End-to-End Context Compression at Scalecited by 4
  13. 2025KVCompose: Efficient Structured KV Cache Compression with Composite Tokenscited by 3
  14. 2026RAP: KV-Cache Compression via RoPE-Aligned Pruningcited by 3
  15. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inferencecited by 3
  16. 2026QCFuse: Query-Aware Cache Fusion via Compressed View for Efficient RAG Servingcited by 3
  17. 2026MemDecay: Region-Aware KV Cache Eviction for Efficient LLM Agent Inferencecited by 3
  18. 2026WitCert: Sound Runtime Risk Observability and Gating for KV-Cache Quantizationcited by 3
  19. 2026RestoreKV: Recovering Full-Cache Behavior Under Aggressive Query-Agnostic KV Cache Evictioncited by 3
  20. 2025Block Sparse Flash Attentioncited by 2
  21. 2026TableCache: Primary Foreign Key Guided KV Cache Precomputation for Low Latency Text-to-SQLcited by 2
  22. 2026ParisKV: Fast and Drift-Robust KV-Cache Retrieval for Long-Context LLMscited by 2
  23. 2026CompilerKV: Risk-Adaptive KV Compression via Offline Experience Compilationcited by 2
  24. 2026Self-Pruned Key-Value Attention: Learning When to Write by Predicting Future Utilitycited by 2
  25. 2026Tangram: Unlocking Non-Uniform KV Cache Compression for Efficient Multi-turn LLM Servingcited by 2
  26. 2026KARA: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compressioncited by 2
  27. 2026Practical Online KV Cache Compaction for LLM Agents: An Empirical Studycited by 2
  28. 2026Does Accuracy Equal Evidence? Reasoning Faithfulness under KV Cache Compressioncited by 2
  29. 2026Cartridges++: KV Cache Compression without Off-Context Derailmentcited by 2
  30. 2025Efficient On-Device Agents via Adaptive Context Managementcited by 1
  31. 2026Free(): Learning to Forget in Malloc-Only Reasoning Modelscited by 1
  32. 2026Transactional Attention: Semantic Sponsorship for KV-Cache Retentioncited by 1
  33. 2026ProxyKV: Cross-Model Proxy Pruning for Efficient Long-Context LLM Inferencecited by 1
  34. 2026Vortex: Efficient and Programmable Sparse Attention Serving for AI Agentscited by 1
  35. 2026Last But Not Least: Boundary Attention CalibratiON for Multimodal KV Cache Compressioncited by 1
  36. 2026DecodeShare: Tracing the Shared Subspace of LLM Decode-Time Decisionscited by 1
  37. 2026LOCKS: Page-Local Compact Key Summaries for Efficient Long-Context Decodingcited by 1
  38. 2026Back from the Future: Key-Value Cache Management by Counter-Causal Surprisecited by 1

50 most cited of 95 All 95 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026C2KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference
  2. 2024KV-Compress: Paged KV-Cache Compression with Variable Compression Rates per Attention Head
  3. 2025KVLink: Accelerating Large Language Models via Efficient KV Cache Reuse
  4. 2025KV Cache Compression for Inference Efficiency in LLMs: A Review
  5. 2025KVComp: A High-Performance, LLM-Aware, Lossy Compression Framework for KV Cache
  6. 2025Compactor: Calibrated Query-Agnostic KV Cache Compression with Approximate Leverage Scores
  7. 2026DeferKV: Rethinking Eviction Timing for One-Shot KV Cache Compression
  8. 2026KV Cache Offloading for Context-Intensive Tasks