mlsanityData to 14 Oct 2026

Protection Is (Nearly) All You Need: Structural Protection Dominates Scoring in Globally Capped KV Eviction

arXiv 2605.18053 ↗First posted 18 May 2026cs.LG cs.CL cs.CRcited by 3

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

We study KV cache eviction under a shared globally capped decode-time harness. Seven policies (LRU, H2O, SnapKV, StreamingLLM, Ada-KV, QUEST, Random) share a prompt-boundary vulnerability: without structural protection, they collapse to near-zero quality on six pure-transformer models (F1≤0.064). Reserving 10% of cache at each boundary recovers 69--90% of the C=2,048 reference-ceiling quality on seven LongBench models at C=256 (13% retention); a ten-model panel spans 68--98%. An attention-mass pilot (Qwen2.5-3B, N=30) suggests why: the position-0 sink holds ~75% of prefix mass, while other boundary tokens sit near ~0.41× uniform expectation, so attention scorers retain the sink but still drop structurally critical tokens. With protection, simplified score-isolation variants are TOST-equivalent to LRU at K=32 (Delta=0.02); at K=8, attention policies pairwise converge yet beat LRU by 0.011--0.021 F1 across C=256 and C=512. Faithful Ada-KV/QUEST add ~0.03--0.04 F1 on Mistral-7B and Phi-3.5 beyond simplified variants. A NIAH-32K regime-transfer pilot on Qwen3-4B (decode vs. prefill, C∈512,2048) shows near-identical protection lifts (ratio 0.99--1.00). At 64K, protection helps but recovery is modest; faithful per-head scoring matches full-cache ceiling on Gemma-3-4B at 6.3% retention only when the model already supports strong 64K retrieval without eviction. Overall: protection dominates; scoring differences are secondary once boundaries are guarded; per-head allocation gives a further modest gain.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 25 of its 33 references that are in the map. First posted on arXiv 18 May 2026.

Builds on

25 of 33 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMs3 Dec
  2. 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Heads22 Jul
  3. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
  4. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  5. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference21 May
  6. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  7. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference14 Mar
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  9. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun
  10. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 15

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024Qwen2.5 Technical Reportcited by 4,942
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  4. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  5. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  7. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  8. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  9. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  10. 2024Yi: Open Foundation Models by 01.AIcited by 719
  11. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  12. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
3 more
  1. 2022Efficiently Scaling Transformer Inferencecited by 539
  2. 2018Competitive caching with machine learned advicecited by 195
  3. 2026ForesightKV: Optimizing KV Cache Eviction for Reasoning Models by Learning Long-Term Contributioncited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Random Attention: Rethinking KV Cache Eviction for Efficient Reasoningcited by 1
  2. 2026Monte Carlo Estimation for KV Cache Evictioncited by 1
  3. 2026PAGE: Partition-Aware Gated KV-Cache Evictioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Risk-Controlled KV-Cache Eviction: From Memory Budgets to Risk Targets
  2. 2026Exploring a Layer-Wise Design Space for KV Cache Eviction
  3. 2026Trust the Mass: Forced Weights in KV-Cache Eviction
  4. 2025Taming the Fragility of KV Cache Eviction in LLM Inference
  5. 2026A Probabilistic Interpretation of KV Cache Eviction
  6. 2026CORE: COverage CAlibration and Evicted-Mass REdistribution for KV Cache
  7. 2026Bit-Flip Vulnerability of Shared KV-Cache Blocks in LLM Serving Systems
  8. 2026LookaheadKV: Fast and Accurate KV Cache Eviction by Glimpsing into the Future without Generation