mlsanityData to 14 Oct 2026

DistillCache: KL-Guided Adaptive KV-Cache Eviction for Memory-Efficient LLM Inference

arXiv 2608.08878 ↗First posted 9 Aug 2026cs.LG cs.AI cs.PFcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Transformer-based large language models (LLMs) achieve strong performance across many tasks, but their Key-Value (KV) cache grows linearly with sequence length, creating a severe memory bottleneck for long-context inference. Existing heuristic eviction methods (e.g., H2O and SnapKV) rely on static attention or positional signals that often fail to capture a token's future predictive influence. We propose DistillCache, a reinforcement learning framework that formulates KV-cache eviction as a sequential decision problem. DistillCache learns a lightweight policy network using rich internal model signals (attention statistics, value norms, entropy, and position) and trains it with REINFORCE via a per-step KL-divergence reward to preserve the full-cache output distribution. On a 7B-parameter instruction-tuned Transformer (Mistral-7B-Instruct-v0.3), DistillCache retains 94.2% of full-cache accuracy on LongBench at a 25% cache budget, outperforming both strong heuristic baselines (H2O, SnapKV) by up to 2.7 absolute points and, under our re-implementations, concurrent RL-based methods (ForesightKV, RLKV) by up to 1.4 points on long-context tasks. On reasoning benchmarks, DistillCache is competitive with the best concurrent method and surpasses it under aggressive compression. It also delivers up to 2.1x full-cache throughput while maintaining competitive practical efficiency. These results highlight the effectiveness of learned, distribution-aware policies for memory-efficient long-context LLM inference.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 12 of its 16 references that are in the map. First posted on arXiv 9 Aug 2026.

Builds on

12 of 16 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environments21 Apr
  2. 2025SpeCache: Speculative Key-Value Caching for Efficient Generation of LLMs20 Mar
  3. 2025CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferences16 Mar
  4. 2025RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression19 Feb
  5. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  6. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference14 Mar
  7. 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference14 Feb
  8. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  9. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 3

  1. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  2. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  3. 2026ForesightKV: Optimizing KV Cache Eviction for Reasoning Models by Learning Long-Term Contributioncited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Reformulating KV Cache Eviction Problem for Long-Context LLM Inference
  2. 2026Learning to Evict from Key-Value Cache
  3. 2024In-context KV-Cache Eviction for LLMs via Attention-Gate
  4. 2024NACL: A General and Effective KV Cache Eviction Framework for LLMs at Inference Time
  5. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  6. 2025G-KV: Decoding-Time KV Cache Eviction with Global Attention
  7. 2026IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inference
  8. 2025LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inference