mlsanityData to 14 Oct 2026

Where Should the KV Cache Live? Placement Policies Across GPU, CPU, and SSD for Long-Lived Sessions

arXiv 2609.16215 ↗First posted 14 Sep 2026cs.AIcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

GPU high bandwidth memory is scarce and expensive, and KV caches consume much of it as chats, agent loops, and document question answering accumulate state. Systems such as Mooncake, LMCache, FlexGen, InfiniGen, and AttentionStore extend GPU memory with CPU DRAM and SSD. The harder question is which blocks belong in each tier, when to move or evict them, and whether prefetching helps. We study these choices in a discrete event simulator spanning GPU HBM, CPU DRAM, and SSD, calibrated against a random forest execution time predictor. We compare recency, reuse frequency, predicted reuse, and an EWMA predictor with prefetch lookahead across chat, agent, and document question answering workloads. Tiering supports 73.02 times more concurrent sessions per GPU and lowers cost per session by 62.04 times. These gains come from tier capacities of 1 plus 8 plus 64, not placement policy. Decode is compute bound at batch size one in our setup, so placement barely affects throughput. It mainly changes PCIe migration traffic and time to first token. Recency produces 2.30 times less migration traffic than reuse frequency for chat. Reuse frequency performs best for agents and document question answering. The existing predicted reuse policy is byte identical to recency, making its agent recommendation effectively recency. A genuine EWMA predictor changes behavior but still ranks behind reuse frequency on the workloads prediction was expected to help. Prefetching does not justify its bandwidth cost. Across the policy and cache size grid, even an oracle with knowledge of future requests never beats no prefetch on migration traffic. Workload specific placement can reduce data movement, but the predicted reuse and prefetch recommendations are not supported as implemented.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 20 of its 25 references that are in the map. First posted on arXiv 14 Sep 2026.

Builds on

20 of 25 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference8 Oct
  2. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  3. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  4. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion26 May
  5. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention23 Mar
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  7. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 13

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  4. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  5. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  6. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  7. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  8. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  9. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  10. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  11. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 196
  12. 2024RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generationcited by 98
1 more
  1. 2024ServerlessLLM: Low-Latency Serverless Inference for Large Language Modelscited by 80

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Predictive Multi-Tier Memory Management for KV Cache in Large-Scale GPU Inference
  2. 2026KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference
  3. 2026Physically Partitioned KVCache Format for CPU--GPU Load Balancing in MoE Inference
  4. 2026More GPUs or a Smaller Cache? Tensor Parallelism versus KV Compression for Memory-Bound LLM Serving
  5. 2026TempoKV: Timely Staging of LLM KV Caches for Memory-Semantic Flash
  6. 2025Strata: Hierarchical Context Caching for Long Context Language Model Serving
  7. 2026The KV Cache Is the New Memory Wall
  8. 2026ScoutAttention: Efficient KV Cache Offloading via Layer-Ahead CPU Pre-computation for LLM Inference