mlsanityData to 7 Oct 2026

KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference

arXiv 2605.18071 ↗First posted 18 May 2026cs.CLcited by 5

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Supporting long-context LLMs is challenging due to the substantial memory demands of the key-value (KV) cache. Existing offloading systems store the full cache in host memory and selectively fetch critical entries during decoding, but this strategy quickly hits a ceiling: sparsity cannot be pushed further without degrading accuracy. As a result, when context length and batch size grow, the volume of KV transfers rises sharply and becomes the dominant source of decoding latency. We present KVDrive, a holistic multi-tier KV cache management system spanning GPU memory, host DRAM, and SSD. Unlike prior work that pursues greater sparsity through algorithmic refinements, KVDrive tackles the problem from a systems perspective - jointly orchestrating cache placement, pipeline scheduling, and cross-tier coordination to sustain high-throughput inference under tight GPU budgets. KVDrive advances three fundamental capabilities: it adapts cache management to attention behavior to maximize reuse and minimize redundant data movement; it restructures the decoding pipeline to overlap I/O- and CPU/GPU compute-bound stages, eliminating stalls across heterogeneous resources; and it harmonizes data movement across memory tiers to unlock scalable long-context inference far beyond GPU and DRAM limits. We have implemented a fully functional prototype of KVDrive and evaluated it on long-context benchmarks with popular LLMs. The system achieves up to 1.74x higher throughput compared to state-of-the-art works while preserving accuracy.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 26 of its 38 references that are in the map. First posted on arXiv 18 May 2026.

Builds on

26 of 38 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025Strata: Hierarchical Context Caching for Long Context Language Model Serving26 Aug
  2. 2025RetroInfer: A Vector Storage Engine for Scalable Long-Context LLM Inference5 May
  3. 2025LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attention20 Feb
  4. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving2 Jan
  5. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference28 Oct
  6. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  7. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  8. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  9. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention23 Mar
  10. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 16

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  3. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  4. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  5. 2024Phi-4 Technical Reportcited by 675
  6. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  7. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
  8. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  9. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 188
  10. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  11. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 121
  12. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 114
4 more
  1. 2025A Comprehensive Survey on Long Context Language Modelingcited by 113
  2. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 112
  3. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUscited by 73
  4. 2020TurboTransformers: An Efficient GPU Serving System For Transformer Modelscited by 71

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Adaptive KV Cache Reuse for Fast Long-Context LLM Servingcited by 2
  2. 2026OasisKV: Scaling In-Decode KV Cache Beyond HBM with Lookahead Sparse Prefetchingcited by 0
  3. 2026Unified AI Gateway: A Framework for Joint Model Routing and KV Cache Managementcited by 0
  4. 2026Physically Partitioned KVCache Format for CPU--GPU Load Balancing in MoE Inferencecited by 0
  5. 2026SPLASH: Co-Designing Sparse Attention with High-Bandwidth Flash for Efficient Long-Context Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026ScoutAttention: Efficient KV Cache Offloading via Layer-Ahead CPU Pre-computation for LLM Inference
  2. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference
  3. 2026CONF-KV: Confidence-Aware KV Cache Eviction with Mixed-Precision Storage for Long-Horizon LLM
  4. 2025Accelerating LLM Inference Throughput via Asynchronous KV Cache Prefetching
  5. 2026TTKV: Temporal-Tiered KV Cache for Long-Context LLM Inference
  6. 2025Understanding Bottlenecks for Efficiently Serving LLM Inference With KV Offloading
  7. 2024MiniKV: Pushing the Limits of LLM Inference via 2-Bit Layer-Discriminative KV Cache
  8. 2026DualDecoder: Accelerate Long Context LLM Inference by Predictive Prefetch