mlsanityData to 14 Oct 2026

When Does Value-Aware KV Eviction Help? A Fixed-Contract Diagnostic for Non-Monotone Cache Compression

arXiv 2605.08234 ↗First posted 7 May 2026cs.LG cs.AIcited by 5

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Long-context LLM inference is bottlenecked by the memory and bandwidth cost of reading large KV caches during decoding. KV compression reduces this cost by keeping only part of the cache, but task accuracy alone does not identify why a selector succeeds or fails. A selector can fail at three steps: it may miss the evidence future decoding needs, give high scores to tokens that do not affect the output, or break related evidence when fitting scores into a small cache. We introduce a fixed-contract diagnostic that holds the selector's setup fixed and changes one decision slot at a time. For value ranking, the probe combines a block's attention mass with the estimated output change from removing it. On LongBench across three models and two budgets, the probe is positive on 72.6% of positive-margin cells and 32.4% of nonpositive-margin cells. NeedleBench M-RT at 32k and a RULER 8k check probe support closure under branched retrieval, and a 264-cell sign evaluation separates support recovery and output-value ranking from leverage effects near the boundary. The resulting order is to recover decode-side evidence, rank its output value, and preserve coupled evidence during projection.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 49 of its 53 references that are in the map. First posted on arXiv 7 May 2026.

Builds on

49 of 53 references.

Closest ancestors · 25

Same direction, newest first

  1. 2026KV Cache Offloading for Context-Intensive Tasks9 Apr
  2. 2026Where Matters More Than What: Decoding-aligned KV Cache Compression via Position-aware Pseudo Queries12 Mar
  3. 2026LookaheadKV: Fast and Accurate KV Cache Eviction by Glimpsing into the Future without Generation11 Mar
  4. 2026ARKV: Adaptive and Resource-Efficient KV Cache Management under Limited Memory Budget for Long-Context Inference in LLMs19 Feb
  5. 2026More Than a Quick Glance: Overcoming the Greedy Bias in KV-Cache Compression2 Feb
  6. 2026Adaptive Layer Selection for Layer-Wise Token Pruning in LLM Inference12 Jan
  7. 2025Taming the Fragility of KV Cache Eviction in LLM Inference15 Oct
  8. 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distribution1 Oct
  9. 2025The Pitfalls of KV Cache Compression30 Sep
  10. 2025GraphKV: Breaking the Static Selection Paradigm with Graph-Based KV Cache Eviction30 Aug
  11. 2025AhaKV: Adaptive Holistic Attention-Driven KV Cache Eviction for Efficient Inference of Large Language Models4 Jun
  12. 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction29 May
13 more
  1. 2025Lookahead Q-Cache: Achieving More Consistent KV Cache Eviction via Pseudo Query24 May
  2. 2025CAOTE: KV Cache Selection for LLMs via Attention Output Error-Based Token Eviction18 Apr
  3. 2025CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferences16 Mar
  4. 2025CriticalKV: Optimizing KV Cache Eviction from an Output Perturbation Perspective6 Feb
  5. 2024Palu: Compressing KV-Cache with Low-Rank Projection30 Jul
  6. 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Heads22 Jul
  7. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
  8. 2024Attention Score is not All You Need for Token Importance Indicator in KV Cache Reduction: Value Also Matters18 Jun
  9. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  10. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  11. 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization28 Feb
  12. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  13. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 24

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  4. 2023Mistral 7Bcited by 3,164
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  6. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  7. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  8. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 1,040
  9. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  10. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  11. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  12. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 407
12 more
  1. 2024Transformers are Multi-State RNNscited by 120
  2. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 116
  3. 2026Attention Residualscited by 65
  4. 2025R-KV: Redundancy-aware KV Cache Compression for Reasoning Modelscited by 54
  5. 2024NeedleBench: Evaluating LLM Retrieval and Reasoning Across Varying Information Densitiescited by 32
  6. 2026IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reusecited by 31
  7. 2026Attention Sink in Transformers: A Survey on Utilization, Interpretation, and Mitigationcited by 11
  8. 2026ForesightKV: Optimizing KV Cache Eviction for Reasoning Models by Learning Long-Term Contributioncited by 7
  9. 2026Do LLMs Encode Functional Importance of Reasoning Tokens?cited by 5
  10. 2025Attention Debiasing for Token Pruning in Vision Language Modelscited by 4
  11. 2026FleetOpt: Analytical Fleet Provisioning for LLM Inference with Compress-and-Route as Implementation Mechanismcited by 3
  12. 2026Beyond Attention Magnitude: Leveraging Inter-layer Rank Consistency for Efficient Vision-Language-Action Modelscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026MomentKV: Closing the Directional Gap in KV Cache Eviction for Long-Context Inferencecited by 6
  2. 2026Error Certificates for KV-Cache Eviction via Randomized Designcited by 2
  3. 2026Trust the Mass: Forced Weights in KV-Cache Evictioncited by 1
  4. 2026PAGE: Partition-Aware Gated KV-Cache Evictioncited by 0
  5. 2026When to Evict, Not What to Keep: Draft-Guided Eviction for Training-Free KV-Cache Compressioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026CORE: COverage CAlibration and Evicted-Mass REdistribution for KV Cache
  2. 2026KVDiagnosis: A Diagnostic Benchmark for KV-Cache Compression in Long-Context Language Models
  3. 2026CONF-KV: Confidence-Aware KV Cache Eviction with Mixed-Precision Storage for Long-Horizon LLM
  4. 2026Behavior-Preserving KV Cache Compression
  5. 2025REAL: REtrieval-reAsoning and Logic-constructed Attention Behaviors for Long-Context KV Cache Compression
  6. 2026ValueDiff: Value-Geometric KV Cache Eviction for Sink-Suppressed LLMs
  7. 2026A Simple Plug-in for Improving Eviction-Based KV Cache Compression
  8. 2026Rethinking KV Cache Eviction via a Unified Information-Theoretic Objective