Risk-Controlled KV-Cache Eviction: From Memory Budgets to Risk Targets
On the map: Efficient LLM inference and serving → KV cache optimization
Abstract (arXiv)
KV-cache eviction is typically evaluated through average quality-memory trade-offs, yet a small average loss can hide requests whose utility degrades materially. We reformulate eviction as a deployment risk-control problem: a material degradation occurs when eviction lowers task utility by more than a deployment-specified tolerance relative to full-KV inference on the same request, and deployment risk is the population frequency of such events. Given a reliability contract specifying a target risk level and confidence requirement, we use a compressor-agnostic post-hoc certification procedure to select a retention policy from calibration data with a finite-sample guarantee, falling back to full KV when no compressed policy is certified. Across multiple eviction methods, Llama and Mistral models, and LongBench and RULER-32K, the same contract supports substantially different levels of eviction: on Llama, it certifies SnapKV at 75% retention on LongBench but no tested compressed policy on RULER-32K, triggering full-KV fallback. Policies with empirical degradation rates below the 5% target can still fail finite-sample certification; on Llama LongBench, empirical thresholding selects uncertified policies that retain 5-10 percentage points less cache across fixed-budget methods. The proposed framework converts a deployment-level reliability requirement into a KV-memory operating point.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 18 of its 26 references that are in the map. First posted on arXiv 23 Sep 2026.
Builds on
18 of 26 references.
Closest ancestors · 7
Same direction, newest first
- 2025Taming the Fragility of KV Cache Eviction in LLM Inference15 Oct
- 2025The Pitfalls of KV Cache Compression30 Sep
- 2025ReFreeKV: Towards Threshold-Free KV Cache Compression24 Feb
- 2024DynamicKV: Task-Aware Adaptive KV Cache Compression for Long Context LLMs19 Dec
- 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
- 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
- 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
Other references · 11
- 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
- 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
- 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
- 2021Distribution-Free, Risk-Controlling Prediction Setscited by 288
- 2021Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Controlcited by 271
- 2024Fast yet Safe: Early-Exiting with Risk Controlcited by 30
- 2022Quantile Risk Control: A Flexible Framework for Bounding the Probability of High-Loss Predictionscited by 14
- 2025Controlling the Risk of Corrupted Contexts for Language Models via Early-Exitingcited by 3
- 2026WitCert: Sound Runtime Risk Observability and Gating for KV-Cache Quantizationcited by 3
- 2026CompilerKV: Risk-Adaptive KV Compression via Offline Experience Compilationcited by 2
- 2026Error Certificates for KV-Cache Eviction via Randomized Designcited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026AvoKV-E: Payload-Aware KV Cache Eviction for Long Reasoning
- 2026A Probabilistic Interpretation of KV Cache Eviction
- 2026Discovering KV Cache Eviction Policies via LLM-Guided Program Evolution
- 2026Rethinking KV Cache Eviction via a Unified Information-Theoretic Objective
- 2026Protection Is (Nearly) All You Need: Structural Protection Dominates Scoring in Globally Capped KV Eviction
- 2026Exploring a Layer-Wise Design Space for KV Cache Eviction
- 2025EVICPRESS: Joint KV-Cache Compression and Eviction for Efficient LLM Serving
- 2026CORE: COverage CAlibration and Evicted-Mass REdistribution for KV Cache