mlsanityData to 14 Oct 2026

KVzap: Fast, Adaptive, and Faithful KV Cache Pruning

arXiv 2601.07891 ↗First posted 12 Jan 2026cs.LG cs.AI cs.CLcited by 15

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Growing context lengths in transformer-based language models have made the key-value (KV) cache a critical inference bottleneck. While many KV cache pruning methods have been proposed, they have not yet been adopted in major inference engines due to speed--accuracy trade-offs. We introduce KVzap, a fast, input-adaptive approximation of KVzip that works in both prefilling and decoding. On Qwen3-8B, Llama-3.1-8B-Instruct, and Qwen3-32B across long-context and reasoning tasks, KVzap achieves 2--4× KV cache compression with negligible accuracy loss and achieves state-of-the-art performance on the KVpress leaderboard. Code and models are available at https://github.com/NVIDIA/kvpress.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 27 of its 40 references that are in the map. First posted on arXiv 12 Jan 2026.

Builds on

27 of 40 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distribution1 Oct
  2. 2025Compactor: Calibrated Query-Agnostic KV Cache Compression with Approximate Leverage Scores10 Jul
  3. 2025Inference-Time Hyper-Scaling with KV Cache Compression5 Jun
  4. 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction29 May
  5. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads14 Oct
  6. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
  7. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  8. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  9. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  10. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  11. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 16

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  3. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  4. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  5. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  6. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  7. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  8. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  9. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 815
  10. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  11. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 360
  12. 2024SeerAttention: Learning Intrinsic Sparse Attention in Your LLMscited by 124
4 more
  1. 2024ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identificationcited by 93
  2. 2024Challenges in Deploying Long-Context Transformers: A Theoretical Peak Performance Analysiscited by 44
  3. 2025Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inferencecited by 33
  4. 2025The Few-shot Dilemma: Over-prompting Large Language Modelscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 15.

15 papers.

Sort:

By area:

  1. 2026Fast KVzip: Efficient and Accurate LLM Inference with Gated KV Evictioncited by 16
  2. 2026Learning to Evict from Key-Value Cachecited by 14
  3. 2026KV Packet: Recomputation-Free Context-Independent KV Caching for LLMscited by 10
  4. 2026KVServe: Service-Aware KV Cache Compression for Communication-Efficient Disaggregated LLM Servingcited by 7
  5. 2026VeriCache: Turning Lossy KV Cache into Lossless LLM Inferencecited by 5
  6. 2026RestoreKV: Recovering Full-Cache Behavior Under Aggressive Query-Agnostic KV Cache Evictioncited by 3
  7. 2026Self-Pruned Key-Value Attention: Learning When to Write by Predicting Future Utilitycited by 2
  8. 2026KV Cache Offloading for Context-Intensive Taskscited by 1
  9. 2026ProxyKV: Cross-Model Proxy Pruning for Efficient Long-Context LLM Inferencecited by 1
  10. 2026EchoPress: Query-Agnostic KV Cache Pruning via Virtual Context Reconstructioncited by 1
  11. 2026KV2: A Self-Refining KV Cachecited by 1
  12. 2026PARTREP: Learning What to Repeat for Decoder-only LLMscited by 0
3 more
  1. 2026Emergent Structure in the Marginal Attention Space of Language Modelscited by 0
  2. 2026Dual-QK: Sharp Queries and Flat Keys for Prunable 2-bit KV Cachescited by 0
  3. 2026Rehearse Everything, Remember Nothing: Attic-KV Rehearses What Will Be Readcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Speedup -- Utilizing KV Cache for Sampling and Reasoning
  2. 2024KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharing
  3. 2025KVComp: A High-Performance, LLM-Aware, Lossy Compression Framework for KV Cache
  4. 2025TaDA: Training-free recipe for Decoding with Adaptive KV Cache Compression and Mean-centering
  5. 2025KVCompose: Efficient Structured KV Cache Compression with Composite Tokens
  6. 2026A JoLT for the KV cache: Near-Lossless KV Cache Compression via Joint Rank-bit Allocation
  7. 2026KVSculpt: KV Cache Compression as Distillation
  8. 2024KV-Compress: Paged KV-Cache Compression with Variable Compression Rates per Attention Head