mlsanityData to 7 Oct 2026

CONF-KV: Confidence-Aware KV Cache Eviction with Mixed-Precision Storage for Long-Horizon LLM

arXiv 2605.24786 ↗First posted 24 May 2026cs.LG cs.AIcited by 2

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Long-horizon LLM inference turns the key--value (KV) cache into the dominant GPU memory consumer and makes per-token attention increasingly expensive. Many common eviction policies use static recency windows or historical attention, leaving unused a signal computed on every decoding step: the model's current uncertainty. We introduce CONF-KV, a KV-cache manager that converts the next-token distribution into a scalar confidence score and uses it to choose the per-step cache budget, retaining more context when the model is uncertain and pruning aggressively when it is confident. Within each budget, tokens are ranked by a composite of accumulated attention mass and recency, while a protected recent window preserves local coherence. We combine the policy with blockwise online-softmax attention, mixed FP16/INT8 storage, and a pyramidal per-layer budget variant. Across four model families and generated lengths up to 4K, CONF-KV stays near the footprint of a fixed 512-token sliding window while remaining within 1.5--2.1 perplexity points of full KV. On Needle-in-a-Haystack up to 32K tokens, CONF-KV reaches 91.4% retrieval accuracy versus 53.8% for sliding windows and 80.6% for H2O; on 75 VisualWebArena tasks it retains 95.3% of full-KV success at 2.8 times lower peak memory.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 23 of its 29 references that are in the map. First posted on arXiv 24 May 2026.

Builds on

23 of 29 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024ZigZagkv: Dynamic KV Cache Compression for Long-context Modeling based on Layer Uncertainty12 Dec
  2. 2024UNComp: Can Matrix Entropy Uncover Sparsity? -- A Compressor Design from an Uncertainty-Aware Perspective4 Oct
  3. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
  4. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  5. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  6. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  7. 2023SparQ Attention: Bandwidth-Efficient LLM Inference8 Dec
  8. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  9. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  10. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun
  11. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 12

  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  2. 2020Longformer: The Long-Document Transformercited by 3,362
  3. 2023Qwen Technical Reportcited by 3,273
  4. 2016Pointer Sentinel Mixture Modelscited by 2,992
  5. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  7. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  8. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  9. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  10. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 359
  11. 2022Confident Adaptive Language Modelingcited by 297
  12. 2026Don't Waste Bits! Adaptive KV-Cache Quantization for Lightweight On-Device LLMscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026MemDecay: Region-Aware KV Cache Eviction for Efficient LLM Agent Inferencecited by 3
  2. 2026Monte Carlo Estimation for KV Cache Evictioncited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference
  2. 2025BaKlaVa -- Budgeted Allocation of KV cache for Long-context Inference
  3. 2026ParisKV: Fast and Drift-Robust KV-Cache Retrieval for Long-Context LLMs
  4. 2025HCAttention: Extreme KV Cache Compression via Heterogeneous Attention Computing for LLMs
  5. 2025Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMs
  6. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  7. 2024MiniKV: Pushing the Limits of LLM Inference via 2-Bit Layer-Discriminative KV Cache
  8. 2025Stateful KV Cache Management for LLMs: Balancing Space, Time, Accuracy, and Positional Fidelity