mlsanityData to 14 Oct 2026

KVShare: An LLM Service System with Efficient and Effective Multi-Tenant KV Cache Reuse

arXiv 2503.16525 ↗First posted 17 Mar 2025cs.CL cs.AIcited by 22

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Recent advances in long-text understanding have pushed the context length of large language models (LLMs) up to one million tokens. It boosts LLMs's accuracy and reasoning capacity but causes exorbitant computational costs and unsatisfactory Time to First Token (TTFT). KV cache reuse, which reuses the exact same KV cache of prefixes and templates or shares similar ones but with extra selective recomputation, offers a promising way to tackle this issue. However, prior studies overlook the cross-request KV reuse and the attention deviations introduced by new tokens during the decoding stage. In this paper, we present a KV cache management module that shares the KV cache across requests under multi-tenant scenarios without sacrificing model accuracy. Our system, KVShare, enables accurate and efficient LLM serving by 1) a Dual-Stage High Deviation algorithm (DHD) that conditionally selects a small portion of KV cache to be recomputed during both prefill and decode phases, and 2) a cache-aware scheduler that prioritizes requests based on their KV cache hit rates and orchestrates continuous batching to achieve enhanced system efficiency and faster TTFT. Multi-task experiments conducted on models such as Qwen2.5-7B,Llama3.1-8B and Yi1.5-9B demonstrate that KVShare reduces TTFT by up to 9.39x and increases 1.2x of the throughput compared to the full KV recompute. Moreover, KVShare achieves 20.38% boost in terms of accuracy compared to SOTA methods.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 14 of its 22 references that are in the map. First posted on arXiv 17 Mar 2025.

Builds on

14 of 22 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  3. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 11

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  3. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  4. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  5. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
  6. 2024Yi: Open Foundation Models by 01.AIcited by 719
  7. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
  8. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 535
  9. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 268
  10. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 196
  11. 2024EPIC: Efficient Position-Independent Caching for Serving Large Language Modelscited by 47

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 15.

22 papers.

Sort:

By area:

  1. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferencecited by 126
  2. 2026ProphetKV: User-Query-Driven Selective Recomputation for Efficient KV Cache Reuse in Retrieval-Augmented Generationcited by 13
  3. 2025MEPIC: Memory Efficient Position Independent Caching for LLM Servingcited by 11
  4. 2025VLCache: Computing 2% Vision Tokens and Reusing 98% for Vision-Language Inferencecited by 7
  5. 2026HYPIC: Accelerating Hybrid-Attention LLM Serving with Position-Independent Cachingcited by 7
  6. 2025Category-Aware Semantic Caching for Heterogeneous LLM Workloadscited by 4
  7. 2025Towards Efficient Agents: A Co-Design of Inference Architecture and Systemcited by 4
  8. 2026OxyGen: Unified KV Cache Management for VLA Inference under Multi-Task Parallelismcited by 4
  9. 2026KV Cache Translation across Heterogeneous Large Language Modelscited by 4
  10. 2025ShadowServe: Interference-Free KV Cache Fetching for Distributed Prefix Cachingcited by 3
  11. 2026LinearKV: One Cached State Suffices for Position-Independent Caching in Hybrid LLMscited by 3
  12. 2026Token-Budget-Aware Pool Routing for Cost-Efficient LLM Inferencecited by 2
10 more
  1. 2026HijackKV: New Threat in Position-Independent KV Cache Reusecited by 2
  2. 2026Burn-After-Use for Preventing Data Leakage through a Secure Multi-Tenant Architecture in Enterprise LLMcited by 1
  3. 2026Dual-Pool Token-Budget Routing for Cost-Efficient and Reliable LLM Servingcited by 1
  4. 2026Contiguity, Not Importance: Budgeted Repair of Stale KV Caches After Document Editscited by 1
  5. 2025HA-RAG: Hotness-Aware RAG Acceleration via Mixed Precision and Data Placementcited by 0
  6. 2026QCFuse: Query-Centric Cache Fusion for Efficient RAG Inferencecited by 0
  7. 2026Fine-Tuning a KV Cache Concatenation-Aware Model or Recomputing KV Caches? Why Not Both?cited by 0
  8. 2026RelaxKV: Recomputation Guided by the Query with Sparse Context Attention for Efficient KV Cache Reusecited by 0
  9. 2026KVCMAS: Efficient KV cache Correction for Shared Context in Multi-Agent Systemscited by 0
  10. 2026ATTUNER: Recomputation-Free KV Cache Reuse via Query-Side Adaptationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Comparative Characterization of KV Cache Management Strategies for LLM Inference
  2. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption
  3. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  4. 2026KV Packet: Recomputation-Free Context-Independent KV Caching for LLMs
  5. 2026The KV Cache Working Set: Online Capacity Planning for LLM Inference Systems
  6. 2026RedKnot: Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention
  7. 2025EFIM: Efficient Serving of LLMs for Infilling Tasks with Improved KV Cache Reuse
  8. 2026C2KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference