mlsanityData to 14 Oct 2026

Fast KVzip: Efficient and Accurate LLM Inference with Gated KV Eviction

arXiv 2601.17668 ↗First posted 25 Jan 2026cs.LG cs.CLcited by 16

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Efficient key-value (KV) cache management is crucial for the practical deployment of large language models (LLMs), yet existing compression techniques often incur a trade-off between performance degradation and computational overhead. We propose a novel gating-based KV cache eviction method for frozen-weight LLMs that achieves high compression ratios with negligible computational cost. Our approach introduces lightweight sink-attention gating modules to identify and retain critical KV pairs, and integrates seamlessly into both the prefill and decoding stages. The proposed gate training algorithm relies on forward passes of an LLM, avoiding expensive backpropagation, while achieving strong task generalization through a task-agnostic reconstruction objective. Extensive experiments across the Qwen2.5-1M, Qwen3, and Gemma3 families show that our method maintains near-lossless performance while evicting up to 70% of the KV cache. The results are consistent across a wide range of tasks, including long-context understanding, code comprehension, and mathematical reasoning, demonstrating the generality of our approach.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 41 of its 49 references that are in the map. First posted on arXiv 25 Jan 2026.

Builds on

41 of 49 references.

Closest ancestors · 14

Same direction, newest first

  1. 2026KVzap: Fast, Adaptive, and Faithful KV Cache Pruning12 Jan
  2. 2025Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMs3 Dec
  3. 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distribution1 Oct
  4. 2025EpiCache: Episodic KV Cache Management for Long-Term Conversation on Resource-Constrained Environments22 Sep
  5. 2025Inference-Time Hyper-Scaling with KV Cache Compression5 Jun
  6. 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction29 May
  7. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methods13 Dec
  8. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads14 Oct
  9. 2024ThinK: Thinner Key Cache by Query-Driven Pruning30 Jul
  10. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
  11. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  12. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
2 more
  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  2. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 27

  1. 2017Attention Is All You Needcited by 60,361
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  6. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  8. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  9. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  10. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  11. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  12. 2025Gemma 3 Technical Reportcited by 1,824
15 more
  1. 2020GLU Variants Improve Transformercited by 1,804
  2. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  3. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  4. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  5. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  6. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  7. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 815
  8. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  9. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  10. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 258
  11. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 198
  12. 2025Qwen2.5-1M Technical Reportcited by 164
  13. 2025LLM-Microscope: Uncovering the Hidden Role of Punctuation in Context Memory of Transformerscited by 12
  14. 2025Which Heads Matter for Reasoning? RL-Guided KV Cache Compressioncited by 10
  15. 2022Xtreme Margin: A Tunable Loss Function for Binary Classification Problemscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 16.

16 papers.

Sort:

By area:

  1. 2026VeriCache: Turning Lossy KV Cache into Lossless LLM Inferencecited by 5
  2. 2026End-to-End Context Compression at Scalecited by 4
  3. 2026RestoreKV: Recovering Full-Cache Behavior Under Aggressive Query-Agnostic KV Cache Evictioncited by 3
  4. 2026Self-Pruned Key-Value Attention: Learning When to Write by Predicting Future Utilitycited by 2
  5. 2026Tangram: Unlocking Non-Uniform KV Cache Compression for Efficient Multi-turn LLM Servingcited by 2
  6. 2026EchoPress: Query-Agnostic KV Cache Pruning via Virtual Context Reconstructioncited by 1
  7. 2026Monte Carlo Estimation for KV Cache Evictioncited by 1
  8. 2026Learning to See What You Need: Gaze Attention for Multimodal Large Language Modelscited by 0
  9. 2026PARTREP: Learning What to Repeat for Decoder-only LLMscited by 0
  10. 2026ImpactHO: Importance-Aware KV Cache Transfer for Multi-User Edge LLM Handovercited by 0
  11. 2026Sigmoid Attention as a Better Substrate for Learned KV Cache Evictioncited by 0
  12. 2026BeaconKV: Key-Value Cache Compression Guided by Beacon Queries for Efficient Large Reasoning Model Inferencecited by 0
4 more
  1. 2026PAGE: Partition-Aware Gated KV-Cache Evictioncited by 0
  2. 2026Emergent Structure in the Marginal Attention Space of Language Modelscited by 0
  3. 2026Generative Adversarial Loopscited by 0
  4. 2026Rehearse Everything, Remember Nothing: Attic-KV Rehearses What Will Be Readcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference
  2. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  3. 2024SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget
  4. 2026C2KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference
  5. 2024In-context KV-Cache Eviction for LLMs via Attention-Gate
  6. 2025HACK: Homomorphic Acceleration via Compression of the Key-Value Cache for Disaggregated LLM Inference
  7. 2026Reformulating KV Cache Eviction Problem for Long-Context LLM Inference
  8. 2025EVICPRESS: Joint KV-Cache Compression and Eviction for Efficient LLM Serving