mlsanityData to 7 Oct 2026

Mitigating KV Cache Competition to Enhance User Experience in LLM Inference

arXiv 2503.13773 ↗First posted 17 Mar 2025cs.CLcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

In Large Language Model (LLM) serving, the KV-cache (KVC) bottleneck causes high tail Time-to-First-Token (TTFT) and Time-Between-Tokens (TBT), impairing user experience, particularly in time-sensitive applications. However, satisfying both TTFT and TBT service-level objectives (SLOs) is challenging. To address this, we propose a system, named CacheOPT for mitigating KV Cache competition, based on key insights from our measurements, incorporating novel components. First, it estimates a request's output length, bounding the deviation with a high specified probability, adjusted based on the request arrival rate. Second, it allocates the estimated KVC demand to a request, and reuses other requests' allocated KVC to avoid preemptions while reducing waiting time. Third, it proactively allocates KVC before instead of at the time a request exhausts its allocation and reserves KVC globally to prevent preemptions. Fourth, it chooses a request that has long TBT SLO, long job remaining time and short preemption time to preempt. Fifth, it selects the shortest-latency strategy between swapping and recomputation for preemptions. Experiments show that CacheOPT achieves up to 3.29× and 2.83× lower tail TBT and tail TTFT, 47% and 53% higher TTFT and TBT SLO attainments, and supports up to 1.58× higher request arrival rate than the state-of-the-art methods.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 18 of its 29 references that are in the map. First posted on arXiv 17 Mar 2025.

Builds on

18 of 29 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  3. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughput9 Jun

Other references · 15

  1. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  2. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  3. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  4. 2024Llumnix: Dynamic Scheduling for Large Language Model Servingcited by 167
  5. 2023Fast Distributed Inference Serving for Large Language Modelscited by 143
  6. 2024DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferencecited by 106
  7. 2024Parrot: Efficient Serving of LLM-based Applications with Semantic Variablecited by 96
  8. 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttentioncited by 93
  9. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipelinecited by 86
  10. 2023Fairness in Serving Large Language Modelscited by 77
  11. 2024ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Cachingcited by 48
  12. 2024ExeGPT: Constraint-Aware Resource Scheduling for LLM Inferencecited by 43
3 more
  1. 2024SampleAttention: Near-Lossless Acceleration of Long Context LLM Inference with Adaptive Structured Sparse Attentioncited by 39
  2. 2024Slice-Level Scheduling for High Throughput and Load Balanced LLM Servingcited by 13
  3. 2024Saving GPU Hours in LLM Inference System Development and Online Workloads with Simulation and DBMS-Inspired Cache Replacement Policiescited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025KVShare: An LLM Service System with Efficient and Effective Multi-Tenant KV Cache Reuse
  2. 2026Comparative Characterization of KV Cache Management Strategies for LLM Inference
  3. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption
  4. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  5. 2026The KV Cache Working Set: Online Capacity Planning for LLM Inference Systems
  6. 2025Towards Efficient Key-Value Cache Management for Prefix Prefilling in LLM Inference
  7. 2026KV Cache Offloading for Context-Intensive Tasks
  8. 2026Characterizing Contention-Induced Reliability Collapse in KV-Cache Timing Side Channels for Multi-Tenant LLM Serving