mlsanityData to 7 Oct 2026

The Pitfalls of KV Cache Compression

arXiv 2510.00231 ↗First posted 30 Sep 2025cs.LG cs.AIcited by 15

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

KV cache compression promises increased throughput and efficiency with negligible loss in performance. While the gains in throughput are indisputable and recent literature has indeed shown minimal degradation on particular benchmarks, in general the consequences of compression in realistic scenarios such as multi-instruction prompting have been insufficiently studied. In this paper, we identify several pitfalls that practitioners should be aware of when deploying KV cache compressed LLMs. We evaluate five KV cache compression methods (StreamingLLM, SnapKV, TOVA, H2O, and K-Norm) on Llama3.1 8B and Qwen2.5 14B under multi-instruction prompting with IFEval. Importantly, we show that certain instructions degrade much more rapidly with compression, effectively causing them to be completely ignored by the LLM. As a practical example, we highlight system prompt leakage as a case study, empirically demonstrating the impact of compression on leakage and general instruction-following. We identify several factors that contribute to system prompt leakage: compression method, instruction order, and KV eviction bias. We then propose simple changes to KV cache eviction policies that can reduce the impact of these factors and improve the overall performance in multi-instruction tasks.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 26 of its 38 references that are in the map. First posted on arXiv 30 Sep 2025.

Builds on

26 of 38 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environments21 Apr
  2. 2025LagKV: Lag-Relative Information of the KV Cache Tells Which Tokens Are Important7 Apr
  3. 2025Q-Filters: Leveraging QK Geometry for Efficient KV Cache Compression4 Mar
  4. 2024A Survey on Large Language Model Acceleration based on KV Cache Management27 Dec
  5. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads14 Oct
  6. 2024ThinK: Thinner Key Cache by Query-Driven Pruning30 Jul
  7. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption25 Jul
  8. 2024KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches1 Jul
  9. 2024A Simple and Effective L2 Norm-Based Strategy for KV Cache Compression17 Jun
  10. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  11. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  12. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 14

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  3. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  4. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  5. 2022Efficiently Scaling Transformer Inferencecited by 529
  6. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  7. 2024PLeak: Prompt Leaking Attacks against Large Language Model Applicationscited by 113
  8. 2023Jailbreaking GPT-4V via Self-Adversarial Attacks with System Promptscited by 62
  9. 2025A Closer Look at System Prompt Robustnesscited by 31
  10. 2025Position is Power: System Prompts as a Mechanism of Bias in Large Language Models (LLMs)cited by 19
  11. 2024Raccoon: Prompt Extraction Benchmark of LLM-Integrated Applicationscited by 16
  12. 2025Semantic Integrity Matters: Benchmarking and Preserving High-Density Reasoning in KV Cache Compressioncited by 16
2 more
  1. 2024A First Look at GPT Apps: Landscape and Vulnerabilitycited by 15
  2. 2024LightTransfer: Your Long-Context LLM is Secretly a Hybrid Model with Effortless Adaptationcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 15.

15 papers.

Sort:

By area:

  1. 2026When Does Value-Aware KV Eviction Help? A Fixed-Contract Diagnostic for Non-Monotone Cache Compressioncited by 4
  2. 2026HARD-KV: Head-Adaptive Regularization for Decoding-time KV Compressioncited by 2
  3. 2026Free(): Learning to Forget in Malloc-Only Reasoning Modelscited by 1
  4. 2026KV Cache Offloading for Context-Intensive Taskscited by 1
  5. 2026Adaptive Filtering of the KV Cache: Diagnosing and Correcting Structural-Role Bias in LLM Inferencecited by 1
  6. 2026KVDiagnosis: A Diagnostic Benchmark for KV-Cache Compression in Long-Context Language Modelscited by 1
  7. 2026Random Attention: Rethinking KV Cache Eviction for Efficient Reasoningcited by 1
  8. 2026Cartridges++: KV Cache Compression without Off-Context Derailmentcited by 1
  9. 2026Monte Carlo Estimation for KV Cache Evictioncited by 1
  10. 2026FreqDepthKV: Frequency-Guided Depth Sharing for Robust KV Cache Compression in Long-Context LLM Inferencecited by 0
  11. 2026DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compressioncited by 0
  12. 2026A Probabilistic Interpretation of KV Cache Evictioncited by 0
3 more
  1. 2026DeltaS: Reading the Gated Linear Attention State for KV Cache Eviction in Streaming Videocited by 0
  2. 2026Risk-Controlled KV-Cache Eviction: From Memory Budgets to Risk Targetscited by 0
  3. 2026When Keywords Drop but Classifiers Hold: Soft Refusals under KV Cache Compressioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Batch-Max: Higher LLM Throughput using Larger Batch Sizes and KV Cache Compression
  2. 2025EVICPRESS: Joint KV-Cache Compression and Eviction for Efficient LLM Serving
  3. 2025Rethinking Key-Value Cache Compression Techniques for Large Language Model Serving
  4. 2026From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving
  5. 2026MosaicKV: Serving Long-Context LLM with Dynamic Two-D KV Cache Compression
  6. 2025AdaptCache: KV Cache Native Storage Hierarchy for Low-Delay and High-Quality Language Model Serving
  7. 2026Discovering KV Cache Eviction Policies via LLM-Guided Program Evolution
  8. 2026PAGE: Partition-Aware Gated KV-Cache Eviction