mlsanityData to 7 Oct 2026

TempoKV: Timely Staging of LLM KV Caches for Memory-Semantic Flash

arXiv 2609.35065 ↗First posted 28 Sep 2026cs.DC cs.AI cs.LGcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization also fits: Serving systems and request scheduling

Abstract (arXiv)

Reusable prefix key-value (KV) caches can outgrow GPU memory in large language model (LLM) serving. A memory-semantic flash hierarchy offers SSD-backed capacity with a limited fast tier, but a logical KV hit is not necessarily ready for GPU retrieval. Demand staging exposes SSD latency, whereas immediate staging can reserve fast-tier capacity long before retrieval begins. We present TempoKV, a timing-aware resource-commitment layer that separates early knowledge of reuse from the acquisition of staging resources. It records reusable-KV hits as metadata-only claims and requests commitment when the runtime-estimated time until retrieval falls to the storage-estimated time needed to make KV resident and protected against eviction. These estimates adapt to runtime progress and staging state, while commitment remains subject to available protected capacity. We implement TempoKV in vLLM and LMCache on an SSD-backed CXL memory device without changing request scheduling. Across two models and three prefix cache ratios, TempoKV reduces protected fast-tier byte-time per request by 63-91% versus immediate staging while retaining much of the serving benefit of advance staging. In a fast-tier capacity sweep, output throughput and p95 time to first token (TTFT) remain nearly unchanged as capacity decreases from 100 to 25 GiB. Compared with unmodified LMCache's Device-DAX L1 configuration, TempoKV reduces p95 TTFT by up to 48.0% and increases output throughput by up to 27.8%.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 23 of its 29 references that are in the map. First posted on arXiv 28 Sep 2026.

Builds on

23 of 29 references.

Closest ancestors · 12

Same direction, newest first

  1. 2026A CXL Memory Rack for Multi-Turn LLM Serving20 Jul
  2. 2026PCR: A Prefetch-Enhanced Cache Reuse System for Low-Latency RAG Serving24 Mar
  3. 2025TraCT: Disaggregated LLM Serving with CXL Shared Memory KV Cache at Rack-Scale20 Dec
  4. 2025Beluga: A CXL-Based Memory Architecture for Scalable and Efficient LLM KVCache Management25 Nov
  5. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference8 Oct
  6. 2025Strata: Hierarchical Context Caching for Long Context Language Model Serving26 Aug
  7. 2025KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Provider3 Jun
  8. 2024Fast State Restoration in LLM Serving with HCache7 Oct
  9. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention23 Mar
  10. 2023Stateful Large Language Model Serving with Pensieve9 Dec
  11. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving11 Oct
  12. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 11

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2024Qwen2.5 Technical Reportcited by 4,713
  3. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  4. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  5. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 200
  6. 2025KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflowscited by 56
  7. 2024Queue management for slo-oriented large language model servingcited by 36
  8. 2025Past-Future Scheduler for LLM Serving under SLA Guaranteescited by 20
  9. 2025SkyByte: Architecting an Efficient Memory-Semantic CXL-based SSD with OS and Hardware Co-designcited by 3
  10. 2026ITME: Inference Tiered Memory Expansion with Disaggregated CXL-Hybrid Memoriescited by 3
  11. 2025MatKV: Trading Compute for Flash Storage in LLM Inferencecited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDs
  2. 2026From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving
  3. 2026The KV Cache Is the New Memory Wall
  4. 2026ObjectCache: Layerwise Object-Storage Retrieval for KV Cache Reuse
  5. 2026KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference
  6. 2026CacheFlow: Efficient LLM Serving with 3D-Parallel KV Cache Restoration
  7. 2026OrbitFlow: SLO-Aware Long-Context LLM Serving with Fine-Grained KV Cache Reconfiguration
  8. 2026PulseInfer: I/O-Centric Sparse KV Cache Offloading for Efficient Long-Context LLM Decoding