PersistentKV: Page-Aware Decode Scheduling for Long-Context LLM Serving on Commodity GPUs
On the map: Efficient LLM inference and serving → KV cache optimization
Abstract (arXiv)
Autoregressive large language model (LLM) serving is increasingly limited by key-value (KV) cache movement rather than dense matrix multiplication. Modern paged-attention systems reduce fragmentation, and mature kernels like FlashInfer provide highly optimized decode attention. However, the best single-kernel implementation is not always the best serving schedule: low-active long-context decode can under-utilize GPUs, while mixed sequence lengths introduce tension between many exact-length launches and coarse padded batches. We present PersistentKV, a native block-table decode attention engine and page-aware scheduling study for grouped-query attention (GQA). PersistentKV maps work by KV-head group, executes directly over native page tables, and adds a compact workqueue schedule executing only non-empty row-KV-head-sequence-split tasks. On an RTX 3060 (FP16, page size 16, Hq=32, Hkv=8, d=128), a calibrated roofline-style policy selects FlashInfer for small active batches, PersistentKV sequence splitting for batch size 1 (B1) long-context steps, and PersistentKV workqueue scheduling for supported B8 long-context GQA steps. With cost-model constants fixed on calibration traces, five held-out seeds improve mean wall decode-token throughput by 1.04x to 1.08x on B8 bimodal, uniform, and Zipf-like workloads, and by 1.40x on a B1 bucketed trace. For the B4 boundary case and uncalibrated GQA ratios, the policy avoids regressions by routing to FlashInfer. We also report an attention-plus-MLP timing proxy and workload counters showing workqueue scheduling reduces launch fan-out from 16.00 to 2.00 launches per step on held-out bimodal B8. These results show that work assignment is a decisive serving-system variable.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 10 of its 14 references that are in the map. First posted on arXiv 25 Jun 2026.
Builds on
10 of 14 references.
Closest ancestors · 2
Same direction, newest first
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
- 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun
Other references · 8
- 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
- 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
- 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
- 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
- 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
- 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
- 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 187
- 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttentioncited by 93
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Minima-KV: Retention-Preserving KV Cache Compression with Mixed-Format Paged Attention
- 2026Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Serving
- 2026Unifying Sparse Attention with Hierarchical Memory for Scalable Long-Context LLM Serving
- 2025FlexiCache: Leveraging Temporal Stability of Attention Heads for Efficient KV Cache Management
- 2026PackInfer: Compute- and I/O-Efficient Attention for Batched LLM Inference
- 2025TinyServe: Query-Aware Cache Selection for Efficient LLM Serving
- 2026Ragged Paged Attention: A High-Performance and Flexible LLM Inference Kernel for TPU
- 2025Paged Attention Meets FlexAttention: Unlocking Long-Context Efficiency in Deployed Inference