mlsanityData to 7 Oct 2026

PEEK: Predictive Queue-Informed KV Cache Management for LLM Serving

arXiv 2607.02525 ↗First posted 10 May 2026cs.DC cs.AIcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

We present PEEK, a lightweight scheduling and eviction framework for both online (streaming) and offline (batch) LLM serving; this paper focuses on the online regime. PEEK maintains an incremental radix tree over the pending queue, exposing prefix-sharing clusters no existing engine surfaces. A low-overhead dual-walk matches the tree against the engine's prefix cache to yield longest-prefix-match for every waiting request; PEEK then admits cluster pioneers first so siblings inherit the freshly cached prefix, a co-designed eviction hook protects blocks ancestral to queued demand, and a multi-lane stride scheduler bounds starvation. On SGLang and vLLM across five workloads up to 4×H100 (DP=2 over TP=2), PEEK delivers up to 3.0×/2.6× cache hit, 7.9×/7.1× TTFT, 6.7×/5.5× E2E, and 3.6×/4.5× throughput gains over each engine's strongest stock baseline (SGLang/vLLM), while matching baselines within noise on workloads with no exploitable prefix structure. Wins hold as KV-cache pressure and inference parallelism scale.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 27 of its 35 references that are in the map. First posted on arXiv 10 May 2026.

Builds on

27 of 35 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference8 Oct
  2. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  3. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion26 May
  4. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  5. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention23 Mar
  6. 2024ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition23 Feb
  7. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving11 Oct
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  9. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun
  10. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 17

  1. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  2. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  3. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  4. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  5. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
  6. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 193
  7. 2024Llumnix: Dynamic Scheduling for Large Language Model Servingcited by 167
  8. 2024RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generationcited by 97
  9. 2024Efficient LLM Scheduling by Learning to Rankcited by 83
  10. 2024Hydragen: High-Throughput LLM Inference with Shared Prefixescited by 71
  11. 2024Preble: Efficient Distributed Prompt Scheduling for LLM Servingcited by 68
  12. 2025Autellix: An Efficient Serving Engine for LLM Agents as General Programscited by 57
5 more
  1. 2025KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflowscited by 56
  2. 2024BatchLLM: Optimizing Large Batched LLM Inference with Global Prefix Sharing and Throughput-oriented Token Batchingcited by 31
  3. 2024Marconi: Prefix Caching for the Era of Hybrid LLMscited by 29
  4. 2025Locality-aware Fair Scheduling in LLM Servingcited by 15
  5. 2025LLM Query Scheduling with Prefix Reuse and Latency Constraintscited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025TokenLake: A Unified Segment-level Prefix Cache Pool for Fine-grained Elastic Long-Context LLM Serving
  2. 2026Position: LLM Serving Needs Mathematical Optimization and Algorithmic Foundations, Not Just Heuristics
  3. 2026DualMap: Enabling Both Cache Affinity and Load Balancing for Distributed LLM Serving
  4. 2026When Fancy Eviction Fails: Rethinking Cache Replacement For LLM Prefix Reuse
  5. 2025KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Provider
  6. 2026From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving
  7. 2026KVServe: Service-Aware KV Cache Compression for Communication-Efficient Disaggregated LLM Serving
  8. 2026PRISM: Fast Online LLM Serving via Scheduling-Memory Co-design