mlsanityData to 7 Oct 2026

CriticalKV: Optimizing KV Cache Eviction from an Output Perturbation Perspective

arXiv 2502.03805 ↗First posted 6 Feb 2025cs.CLcited by 28

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Large language models have revolutionized natural language processing but face significant challenges of high storage and runtime costs, due to the transformer architecture's reliance on self-attention, particularly the large KV cache for long-sequence inference. Recent efforts to reduce KV cache size by pruning less critical entries based on attention weights remain empirical and lack formal grounding. This paper presents a formal study on identifying critical KV cache entries by analyzing attention output perturbation. Our analysis reveals that, beyond attention weights, the value states within KV entries and pretrained parameter matrices are also crucial. Based on this, we propose a perturbation-constrained selection algorithm that optimizes the worst-case output perturbation to identify critical entries. We demonstrate that our algorithm is a universal, plug-and-play enhancement that incurs negligible computational overhead. When integrated with three state-of-the-art cache eviction methods on three distinct LLMs, our algorithm significantly reduces the compression loss by more than textithalf on average across 29 datasets from the Ruler and LongBench benchmarks. Further perturbation analysis, at both the head and layer levels, confirms the principles underlying our effectiveness. This work offers a new, formally grounded perspective to cache eviction , opening promising avenues for future research. The code is publicly available at https://github.com/FFY0/DefensiveKV.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 41 of its 56 references that are in the map. First posted on arXiv 6 Feb 2025.

Builds on

41 of 56 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methods13 Dec
  2. 2024Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning25 Oct
  3. 2024KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharing24 Oct
  4. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads14 Oct
  5. 2024ThinK: Thinner Key Cache by Query-Driven Pruning30 Jul
  6. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
  7. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  8. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  9. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Models23 May
  10. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference21 May
  11. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  12. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference14 Mar
3 more
  1. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  2. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun
  3. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 26

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  4. 2023Mistral 7Bcited by 3,065
  5. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  6. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  7. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  8. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  9. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  10. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  11. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
  12. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
14 more
  1. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 893
  2. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  3. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  4. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  5. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  6. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
  7. 2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 431
  8. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 363
  9. 2021Efficient Attentions for Long Document Summarizationcited by 342
  10. 2020Understanding the Difficulty of Training Transformerscited by 210
  11. 2024A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systemscited by 170
  12. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decodingcited by 98
  13. 2024LightTransfer: Your Long-Context LLM is Secretly a Hybrid Model with Effortless Adaptationcited by 11
  14. 2024CritiPrefill: A Segment-wise Criticality-based Approach for Prefilling Acceleration in LLMscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 22.

28 papers.

Sort:

By area:

  1. 2026A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agentscited by 40
  2. 2025SpecVLM: Enhancing Speculative Decoding of Video LLMs via Verifier-Guided Token Pruningcited by 19
  3. 2025LAVa: Layer-wise KV Cache Eviction with Dynamic Budget Allocationcited by 16
  4. 2025Taming the Fragility of KV Cache Eviction in LLM Inferencecited by 14
  5. 2025MixKVQ: Query-Aware Mixed-Precision KV Cache Quantization for Long-Context Reasoningcited by 8
  6. 2026LCGuard: Latent Communication Guard for Safe KV Sharing in Multi-Agent Systemscited by 6
  7. 2026From Prefix Cache to Fusion RAG Cache: Accelerating LLM Inference in Retrieval-Augmented Generationcited by 5
  8. 2025Beyond RAG: Task-Aware KV Cache Compression for Comprehensive Knowledge Reasoningcited by 4
  9. 2026Rethinking KV Cache Eviction via a Unified Information-Theoretic Objectivecited by 4
  10. 2026LKV: End-to-End Learning of Head-wise Budgets and Token Selection for LLM KV Cache Evictioncited by 4
  11. 2026When Does Value-Aware KV Eviction Help? A Fixed-Contract Diagnostic for Non-Monotone Cache Compressioncited by 4
  12. 2026RDKV: Rate-Distortion Bit Allocation for Joint Eviction and Quantization of the KV Cachecited by 4
16 more
  1. 2026ManifoldKV: Training-Free KV Cache Compression via Euclidean Outlier Detectioncited by 2
  2. 2025Neural Attention Searchcited by 1
  3. 2026Sublinear Time Quantum Algorithm for Attention Approximationcited by 1
  4. 2026Reformulating KV Cache Eviction Problem for Long-Context LLM Inferencecited by 1
  5. 2026KV Cache Compression Through the Lens of Transform Codingcited by 1
  6. 2026Trust the Mass: Forced Weights in KV-Cache Evictioncited by 1
  7. 2026T-LoopFormer: Token-Level Elastic-Depth Looped Transformers for Latent Reasoning with Dynamic Routingcited by 1
  8. 2026Monte Carlo Estimation for KV Cache Evictioncited by 1
  9. 2026Head-Aware Key-Value Compression for Efficient Autoregressive Image Generationcited by 0
  10. 2026AnchorKV: Anchor-Residual KV Cache Compressioncited by 0
  11. 2026Parser States Already Know: Structure-Conditioned KV Persistence for Structured Generationcited by 0
  12. 2026From Token Importance to Conditional Removability: Rethinking Visual Token Pruning in Multimodal Large Language Modelscited by 0
  13. 2026When to Evict, Not What to Keep: Draft-Guided Eviction for Training-Free KV-Cache Compressioncited by 0
  14. 2026ReSS: Residual-Restoring Sparse Attention for 3D Vision Transformerscited by 0
  15. 2026OVAL: Output-Aware Local Page Bases for KV Cache Retrievalcited by 0
  16. 2026VisionWeave: Weaving Elastic Visual Representations as a Native Capability of MLLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026DeferKV: Rethinking Eviction Timing for One-Shot KV Cache Compression
  2. 2024CORM: Cache Optimization with Recent Message for Large Language Model Inference
  3. 2025KV Cache Compression for Inference Efficiency in LLMs: A Review
  4. 2025KVSink: Understanding and Enhancing the Preservation of Attention Sinks in KV Cache Quantization for LLMs
  5. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  6. 2025CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferences
  7. 2026ReST-KV: Robust KV Cache Eviction with Layer-wise Output Reconstruction and Spatial-Temporal Smoothing
  8. 2025PackKV: Reducing KV Cache Memory Footprint through LLM-Aware Lossy Compression