mlsanityData to 7 Oct 2026

LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference

arXiv 2510.09665 ↗First posted 8 Oct 2025cs.LGcited by 119

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

KV cache has traditionally been stored in GPU memory to accelerate the decoding phase of large language model (LLM) inference. However, it is increasingly necessary to move KV caches outside GPU devices, to enable cache reuse across different queries and inference engines. Our real-world usage statistics confirm this trend: over time, the total KV cache stored by users has grown rapidly, far exceeding the capacity of GPU memory. Despite this need, there lacks an efficient solution for offloading and transferring KV caches. We present LMCACHE, the first and so far the most efficient open-source KV caching solution, which extracts and stores KV caches generated by modern LLM engines (vLLM and SGLang) out of the GPU memory and shares them across engines and queries. LMCACHE supports both cache offloading (prefix reuse across queries) and prefill-decode (PD) disaggregation (cross-engine/GPU cache transfer). LMCACHE's high performance and wide adoption stem from the following contributions: (1) highly optimized KV cache data movement powered by batched data movement operations, compute and I/O pipelining; (2) a modular KV cache connector component, decoupling LMCACHE from the rapid evolution of inference engines; (3) a first-class control API for flexible cache orchestration across GPU, CPU, storage, and network layers. Our evaluation shows that combining LMCACHE with vLLM achieves up to 15x improvement in throughput across workloads such as multi-round question answering and document analysis. Large-scale adoption of LMCACHE in enterprise settings provides us valuable insights, for example, fetching KV cache from remote storage has unsurprisingly benefits to prefill delay, and that context truncation, which is a widely applied technique in industry, can greatly reduce prefix cache hit ratio by half. The source code of LMCACHE is at: https://github.com/LMCache/LMCache.

Key facts. Cited by 119 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 29 of its 55 references that are in the map. First posted on arXiv 8 Oct 2025.

Builds on

29 of 55 references.

Closest ancestors · 14

Same direction, newest first

  1. 2025Strata: Hierarchical Context Caching for Long Context Language Model Serving26 Aug
  2. 2025BitDecoding: Unlocking Tensor Cores for Long-Context LLMs with Low-Bit KV Cache24 Mar
  3. 2025KVShare: An LLM Service System with Efficient and Effective Multi-Tenant KV Cache Reuse17 Mar
  4. 2025CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferences16 Mar
  5. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads14 Oct
  6. 2024Compute Or Load KV Cache? Why Not Both?4 Oct
  7. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  8. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  9. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention23 Mar
  10. 2024ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition23 Feb
  11. 2023Stateful Large Language Model Serving with Pensieve9 Dec
  12. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving11 Oct
2 more
  1. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 15

  1. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  2. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  3. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  4. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  5. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  6. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  7. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
  8. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 193
  9. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 93
  10. 2025CommVQ: Commutative Vector Quantization for KV Cache Compressioncited by 25
  11. 2024BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batchingcited by 19
  12. 2025AIBrix: Towards Scalable, Cost-Effective Large Language Model Inference Infrastructurecited by 12
3 more
  1. 2025Nexus:Proactive Intra-GPU Disaggregation of Prefill and Decode in LLM Servingcited by 12
  2. 2024Aladdin: Joint Placement and Scaling for SLO-Aware LLM Servingcited by 11
  3. 2025An Extensible Software Transport Layer for GPU Networkingcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 13; 2026: 106.

119 papers.

Sort:

By area:

  1. 2025Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Livecited by 43
  2. 2025KVCOMM: Online Cross-context KV-cache Communication for Efficient LLM-based Multi-agent Systemscited by 35
  3. 2026Don't Break the Cache: An Evaluation of Prompt Caching for Long-Horizon Agentic Taskscited by 18
  4. 2026ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference Systemcited by 17
  5. 2025KV Cache Transform Coding for Compact Storage in LLM Inferencecited by 11
  6. 2025MEPIC: Memory Efficient Position Independent Caching for LLM Servingcited by 10
  7. 2026KV Packet: Recomputation-Free Context-Independent KV Caching for LLMscited by 10
  8. 2025TokenScale: Timely and Accurate Autoscaling for Disaggregated LLM Serving with Token Velocitycited by 9
  9. 2025EVICPRESS: Joint KV-Cache Compression and Eviction for Efficient LLM Servingcited by 9
  10. 2026Agentic Coding in the Wild: Characterizing GitHub Copilot Traces at Production Scalecited by 9
  11. 2025ContextPilot: Fast Long-Context Inference via Context Reusecited by 8
  12. 2026LLMServingSim 2.0: A Unified Simulator for Heterogeneous and Disaggregated LLM Serving Infrastructurecited by 8
38 more
  1. 2026Agent Memory Below the Prompt: Persistent Q4 KV Cache for Multi-Agent LLM Inference on Edge Devicescited by 8
  2. 2026CacheWise: Understanding Workloads and Optimizing KVCache Management for Efficiently Serving LLM Coding Agentscited by 8
  3. 2026KVServe: Service-Aware KV Cache Compression for Communication-Efficient Disaggregated LLM Servingcited by 7
  4. 2026Pythia: Exploiting Workflow Predictability for Efficient Agent-Native LLM Servingcited by 6
  5. 2026Revati: Transparent GPU-Free Time-Warp Emulation for LLM Servingcited by 5
  6. 2026Sutradhara: An Intelligent Orchestrator-Engine Co-design for Tool-based Agentic Inferencecited by 5
  7. 2026Native LLM and MLLM Inference at Scale on Apple Siliconcited by 5
  8. 2026Hardware Co-Design Scaling Laws via Roofline Modelling for On-Device LLMscited by 5
  9. 2026Self-Improvement of Large Language Models: A Technical Overview and Future Outlookcited by 5
  10. 2026VeriCache: Turning Lossy KV Cache into Lossless LLM Inferencecited by 5
  11. 2026Leyline: KV Cache Directives for Agentic Inferencecited by 5
  12. 2025Tree Training: Accelerating Agentic LLMs Training via Shared Prefix Reusecited by 4
  13. 2025FlexiCache: Leveraging Temporal Stability of Attention Heads for Efficient KV Cache Managementcited by 4
  14. 2025Towards Efficient Agents: A Co-Design of Inference Architecture and Systemcited by 4
  15. 2026PCR: A Prefetch-Enhanced Cache Reuse System for Low-Latency RAG Servingcited by 4
  16. 2026Unifying Sparse Attention with Hierarchical Memory for Scalable Long-Context LLM Servingcited by 4
  17. 2026Efficient Serving for Dynamic Agent Workflows with Prediction-based KV-Cache Managementcited by 4
  18. 2025Whose Narrative is it Anyway? A KV Cache Manipulation Attackcited by 3
  19. 2026PrefillShare: A Shared Prefill Module for KV Reuse in Multi-LLM Disaggregated Servingcited by 3
  20. 2026CodecSight: Leveraging Video Codec Signals for Efficient Streaming VLM Inferencecited by 3
  21. 2026PolyKV: A Shared Asymmetrically-Compressed KV Cache Pool for Multi-Agent LLM Inferencecited by 3
  22. 2026An Efficient Hybrid Sparse Attention with CPU-GPU Parallelism for Long-Context Inferencecited by 3
  23. 2026A Policy-Driven Runtime Layer for Agentic LLM Servingcited by 3
  24. 2026Characterization of Multi-Model Agentic AI Systems on General Tasks via Trace-Driven Simulationcited by 3
  25. 2026RaBitQCache: Rotated Binary Quantization for KVCache in Long Context LLM Inferencecited by 3
  26. 2025LiteCache: A Query Similarity-Driven, GPU-Centric KVCache Subsystem for Efficient LLM Inferencecited by 2
  27. 2026DUAL-BLADE: Dual-Path NVMe-Direct KV-Cache Offloading for Edge LLM Inferencecited by 2
  28. 2026Adaptive KV Cache Reuse for Fast Long-Context LLM Servingcited by 2
  29. 2026Observation, Not Prediction: Conversation-Level Disaggregated Scheduling for Agentic Servingcited by 2
  30. 2026QCFuse: Query-Aware Cache Fusion via Compressed View for Efficient RAG Servingcited by 2
  31. 2026MiniPIC: Flexible Position-Independent Caching in <100LOCcited by 2
  32. 2026SwiftCache: Efficient LLM Serving for Multi-turn Conversations with Heterogeneous KV Cache Sharingcited by 2
  33. 2026Agent-Assisted Side-Channel Attacks on Non-Prefix KV Cache in RAGcited by 2
  34. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimizationcited by 2
  35. 2026Nalar: An agent serving frameworkcited by 1
  36. 2026Parallel Context-of-Experts Decoding for Retrieval Augmented Generationcited by 1
  37. 2025Understanding Bottlenecks for Efficiently Serving LLM Inference With KV Offloadingcited by 1
  38. 2026SLO-Aware Compute Resource Allocation for Prefill-Decode Disaggregated LLM Inferencecited by 1

50 most cited of 119 All 119 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  2. 2026KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference
  3. 2026Comparative Characterization of KV Cache Management Strategies for LLM Inference
  4. 2024XKV: Personalized KV Cache Memory Reduction for Long-Context LLM Inference
  5. 2026IceCache: Memory-efficient KV-cache Management for Long-Sequence LLMs
  6. 2025Accelerating LLM Inference Throughput via Asynchronous KV Cache Prefetching
  7. 2024KVPR: Efficient LLM Inference with I/O-Aware KV Cache Partial Recomputation
  8. 2025WindowKV: Task-Adaptive Group-Wise KV Cache Window Selection for Efficient LLM Inference