mlsanityData to 7 Oct 2026

Queue management for slo-oriented large language model serving

arXiv 2407.00047 ↗First posted 5 Jun 2024cs.DC cs.CL cs.LGcited by 36

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Large language model (LLM) serving is becoming an increasingly critical workload for cloud providers. Existing LLM serving systems focus on interactive requests, such as chatbots and coding assistants, with tight latency SLO requirements. However, when such systems execute batch requests that have relaxed SLOs along with interactive requests, it leads to poor multiplexing and inefficient resource utilization. To address these challenges, we propose QLM, a queue management system for LLM serving. QLM maintains batch and interactive requests across different models and SLOs in a request queue. Optimal ordering of the request queue is critical to maintain SLOs while ensuring high resource utilization. To generate this optimal ordering, QLM uses a Request Waiting Time (RWT) Estimator that estimates the waiting times for requests in the request queue. These estimates are used by a global scheduler to orchestrate LLM Serving Operations (LSOs) such as request pulling, request eviction, load balancing, and model swapping. Evaluation on heterogeneous GPU devices and models with real-world LLM serving dataset shows that QLM improves SLO attainment by 40-90% and throughput by 20-400% while maintaining or improving device utilization compared to other state-of-the-art LLM serving systems. QLM's evaluation is based on the production requirements of a cloud provider. QLM is publicly available at https://www.github.com/QLM-project/QLM.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 21 of its 55 references that are in the map. First posted on arXiv 5 Jun 2024.

Builds on

21 of 55 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Services25 Apr
  2. 2024Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction12 Apr
  3. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  4. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instances27 Nov

Other references · 17

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  4. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  5. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  6. 2016Clipper: A Low-Latency Online Prediction Serving Systemcited by 191
  7. 2022Can Foundation Models Wrangle Your Data?cited by 173
  8. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 146
  9. 2020Serving DNNs like Clockwork: Performance Predictability from the Bottom Upcited by 119
  10. 2024A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methodscited by 101
  11. 2017Recent Advances in Recurrent Neural Networkscited by 89
  12. 2017TensorFlow-Serving: Flexible, High-Performance ML Servingcited by 85
5 more
  1. 2020TurboTransformers: An Efficient GPU Serving System For Transformer Modelscited by 71
  2. 2020FIRM: An Intelligent Fine-Grained Resource Management Framework for SLO-Oriented Microservicescited by 41
  3. 2024LAB: Large-Scale Alignment for ChatBotscited by 39
  4. 2024RelayAttention for Efficient Large Language Model Serving with Long System Promptscited by 13
  5. 2019Perseus: Characterizing Performance and Cost of Multi-Tenant Serving for CNN Modelscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 17; 2026: 15.

36 papers.

Sort:

By area:

  1. 2025Prism: Cost-Efficient Multi-LLM Serving via GPU Memory Ballooningcited by 23
  2. 2025Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUscited by 21
  3. 2024SLO-aware GPU Frequency Scaling for Energy Efficient LLM Inference Servingcited by 18
  4. 2025ModServe: Modality- and Stage-Aware Resource Disaggregation for Scalable Multimodal Model Servingcited by 18
  5. 2025AdaSpec: Adaptive Speculative Decoding for Fast, SLO-Aware Large Language Model Servingcited by 18
  6. 2025Throughput-Optimal Scheduling Algorithms for LLM Inference and AI Agentscited by 17
  7. 2025Taming the Titans: A Survey of Efficient LLM Inference Servingcited by 16
  8. 2025AIBrix: Towards Scalable, Cost-Effective Large Language Model Inference Infrastructurecited by 12
  9. 2025Nexus:Proactive Intra-GPU Disaggregation of Prefill and Decode in LLM Servingcited by 12
  10. 2025HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-locationcited by 10
  11. 2024Multi-Bin Batching for Increasing LLM Inference Throughputcited by 9
  12. 2025ElasticMM: Efficient Multimodal LLMs Serving with Elastic Multimodal Parallelismcited by 7
24 more
  1. 2025Scorpio: Serving Right Requests at the Right Time for Heterogeneous SLOs in LLM Inferencecited by 6
  2. 2026OrbitFlow: SLO-Aware Long-Context LLM Serving with Fine-Grained KV Cache Reconfigurationcited by 6
  3. 2024FastSwitch: Optimizing Context Switching Efficiency in Fairness-aware Large Language Model Servingcited by 5
  4. 2025FineServe: Precision-Aware KV Slab and Two-Level Scheduling for Heterogeneous Precision LLM Servingcited by 3
  5. 2026Towards Resiliency in Large Language Model Serving with KevlarFlowcited by 3
  6. 2026FlowPrefill: Decoupling Preemption from Prefill Scheduling Granularity to Mitigate Head-of-Line Blocking in LLM Servingcited by 3
  7. 2025AGFT: An Adaptive GPU Frequency Tuner for Real-Time LLM Inference Optimizationcited by 2
  8. 2025Astrolabe: Balancing Load in LLM Serving with Randomized Prediction-Guided Schedulingcited by 2
  9. 2025WarmServe: Enabling One-for-Many GPU Prewarming for Multi-LLM Servingcited by 2
  10. 2025RepetitionCurse: Measuring and Understanding Router Imbalance in Mixture-of-Experts LLMs under DoS Stresscited by 2
  11. 2026SuperInfer: SLO-Aware Rotary Scheduling and Memory Management for LLM Inference on Superchipscited by 2
  12. 2024TimelyLLM: Segmented LLM Serving System for Time-sensitive Robotic Applicationscited by 1
  13. 2025Efficient Serving of LLM Applications with Probabilistic Demand Modelingcited by 1
  14. 2026PASCAL: A Phase-Aware Scheduling Algorithm for Serving Reasoning-based Large Language Modelscited by 1
  15. 2026RouteBalance: Fused Model Routing and Load Balancing for Heterogeneous LLM Servingcited by 1
  16. 2026SAIR: Cost-Efficient Multi-Stage ML Pipeline Autoscaling via In-Context Reinforcement Learningcited by 0
  17. 2026DharmaOCR: Specialized Small Language Models for Structured OCR that outperform Open-Source and Commercial Baselinescited by 0
  18. 2026HexAGenT: Efficient Agentic LLM Serving via Workflow- and Heterogeneity-Aware Schedulingcited by 0
  19. 2026Beyond Prediction: Tail-Aware Scheduling for LLM Inferencecited by 0
  20. 2026Energy-Aware Scheduling for Serverless LLM Serving on Shared GPUscited by 0
  21. 2026Beyond Accuracy and Cost: Latency-Aware LLM Query Routing for Dynamic Workloadscited by 0
  22. 2026Crossflow: Prefill-Decode Elasticity for Agentic LLM Servingcited by 0
  23. 2026Cross-Model Autoscaling for Shared LLM Servingcited by 0
  24. 2026TempoKV: Timely Staging of LLM KV Caches for Memory-Semantic Flashcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025SLOs-Serve: Optimized Serving of Multi-SLO LLMs
  2. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving
  3. 2025Hierarchical Autoscaling for Large Language Model Serving with Chiron
  4. 2025ProServe: Unified Multi-Priority Request Scheduling for LLM Serving
  5. 2026Beyond Binary Priorities: Multi-Tier SLA Scheduling for Large Language Model Serving
  6. 2025Niyama : Breaking the Silos of LLM Inference Serving
  7. 2025HFX: Joint Design of Algorithms and Systems for Multi-SLO Serving and Fast Scaling
  8. 2026Taming Request Imbalance: SLO-Aware Scheduling for Disaggregated LLM Inference ↗