mlsanityData to 7 Oct 2026

A Queueing-Theoretic Framework for Stability Analysis of LLM Inference with KV Cache Memory Constraints

arXiv 2605.04595 ↗First posted 6 May 2026cs.LG cs.AI math.OCcited by 2

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

The rapid adoption of large language models (LLMs) has created significant challenges for efficient inference at scale. Unlike traditional workloads, LLM inference is constrained by both computation and the memory overhead of key-value (KV) caching, which accelerates decoding but quickly exhausts GPU memory. In this paper, we introduce the first queueing-theoretic framework that explicitly incorporates both computation and GPU memory constraints into the analysis of LLM inference. Based on this framework, we derive rigorous stability and instability conditions that determine whether an LLM inference service can sustain incoming demand without unbounded queue growth. This result offers a powerful tool for system deployment, potentially addressing the core challenge of GPU provisioning. By combining an estimated request arrival rate with our derived stable service rate, operators can calculate the necessary cluster size to avoid both costly over-purchasing and performance-violating under-provisioning. We further validate our theoretical predictions through extensive experiments in real GPU production environments. Our results show that the predicted stability conditions are highly accurate, with deviations typically within 10%.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 24 of its 42 references that are in the map. First posted on arXiv 6 May 2026.

Builds on

24 of 42 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025LLM Serving Optimization with Variable Prefill and Decode Lengths8 Aug
  2. 2025Throughput-Optimal Scheduling Algorithms for LLM Inference and AI Agents10 Apr
  3. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  4. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills31 Aug
  5. 2023Fast Distributed Inference Serving for Large Language Models10 May

Other references · 19

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020Scaling Laws for Neural Language Modelscited by 6,411
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  6. 2022Emergent Abilities of Large Language Modelscited by 2,483
  7. 2022Efficiently Scaling Transformer Inferencecited by 529
  8. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 367
  9. 2024ORLM: A Customizable Framework in Training Large Models for Automated Optimization Modelingcited by 124
  10. 2024Long-Context Language Modeling with Parallel Context Encodingcited by 84
  11. 2022Parallel Context Windows for Large Language Modelscited by 81
  12. 2024Training-Free Long-Context Scaling of Large Language Modelscited by 61
7 more
  1. 2025Online Scheduling for LLM Inference with KV Cache Constraintscited by 17
  2. 2024A Queueing Theoretic Perspective on Low-Latency LLM Inference with Variable Token Lengthcited by 16
  3. 2025Optimizing LLM Inference: Fluid-Guided Online Scheduling with Memory Constraintscited by 16
  4. 2024Multi-Bin Batching for Increasing LLM Inference Throughputcited by 9
  5. 2025Adaptively Robust LLM Inference Optimization under Prediction Uncertaintycited by 8
  6. 2026A Universal Load Balancing Principle and Its Application to Large Language Model Servingcited by 6
  7. 2025Queueing, Predictions, and LLMs: Challenges and Open Problemscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Anatomy of a Quantized Agent: VRAM Stability and Forecasting in Code-Synthesis Agentic Workloadscited by 0
  2. 2026Analytical Power-Aware Provisioning for Prefill-Decode Disaggregated AI Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Efficient LLM Inference with Activation Checkpointing and Hybrid Caching
  2. 2024AQUA: Network-Accelerated Memory Offloading for LLMs in Scale-Up GPU Domains
  3. 2025LLaMCAT: Optimizing Large Language Model Inference with Cache Arbitration and Throttling
  4. 2024Saving GPU Hours in LLM Inference System Development and Online Workloads with Simulation and DBMS-Inspired Cache Replacement Policies
  5. 2024KVDirect: Distributed Disaggregated LLM Inference
  6. 2026Flow-Controlled Scheduling for LLM Inference with Provable Stability Guarantees
  7. 2025Parallax: Efficient LLM Inference Service over Decentralized Environment
  8. 2026Service-Induced Congestion in Memory-Constrained LLM Serving