Aladdin: Joint Placement and Scaling for SLO-Aware LLM Serving
On the map: Efficient LLM inference and serving → Serving systems and request scheduling
Abstract (arXiv)
The demand for large language model (LLM) inference is gradually dominating the artificial intelligence workloads. Therefore, there is an urgent need for cost-efficient inference serving. Existing work focuses on single-worker optimization and lacks consideration of cluster-level management for both inference queries and computing resources. However, placing requests and managing resources without considering the query features easily causes SLO violations or resource underutilization. Providers are forced to allocate extra computing resources to guarantee user experience, leading to additional serving costs. In this paper we introduce Aladdin, a scheduler that co-adaptively places queries and scales computing resources with SLO awareness. For a stream of inference queries, Aladdin first predicts minimal computing resources and the corresponding serving workers' configuration required to fulfill the SLOs for all queries. Then, it places the queries to each serving worker according to the prefill and decode latency models of batched LLM inference to maximize each worker's utilization. Results show that Aladdin reduces the serving cost of a single model by up to 71% for the same SLO level compared with the baselines, which can be millions of dollars per year.
Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 12 of its 27 references that are in the map. First posted on arXiv 11 May 2024.
Builds on
12 of 27 references.
Closest ancestors · 4
Same direction, newest first
- 2024Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction12 Apr
- 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
- 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
- 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline22 May
Other references · 8
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
- 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
- 2016Clipper: A Low-Latency Online Prediction Serving Systemcited by 191
- 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 121
- 2020Serving DNNs like Clockwork: Performance Predictability from the Bottom Upcited by 120
- 2024D'ej`aVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Servingcited by 48
- 2024ExeGPT: Constraint-Aware Resource Scheduling for LLM Inferencecited by 45
Built on by
Citing papers per year: 2024: 2; 2025: 8; 2026: 1.
11 papers.
- 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferencecited by 126
- 2024A Survey on Large Language Models for Communication, Network, and Service Management: Application Insights, Challenges, and Future Directionscited by 25
- 2024Closer Look at Efficient Inference Methods: A Survey of Speculative Decodingcited by 7
- 2026DualScale: Energy-Efficient Disaggregated LLM Serving via Phase-Aware Placement and DVFScited by 5
- 2025HFX: Joint Design of Algorithms and Systems for Multi-SLO Serving and Fast Scalingcited by 3
- 2025Cost-Efficient LLM Serving in the Cloud: VM Selection with KV Cache Offloadingcited by 2
- 2025Rethinking Caching for LLM Serving Systems: Beyond Traditional Heuristicscited by 2
- 2025Efficient Serving of LLM Applications with Probabilistic Demand Modelingcited by 1
- 2025SageServe: Optimizing LLM Serving on Cloud Data Centers with Forecast Aware Auto-Scalingcited by 0
- 2025A Predictive and Synergistic Two-Layer Scheduling Framework for LLM Servingcited by 0
- 2025BeLLMan: Controlling LLM Congestioncited by 0
- 2026DualScale: Energy-Efficient Disaggregated LLM Serving via Phase-Aware Placement and DVFS21 Feb 2026
- 2025BeLLMan: Controlling LLM Congestion17 Oct 2025
- 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference8 Oct 2025
- 2025A Predictive and Synergistic Two-Layer Scheduling Framework for LLM Serving27 Sep 2025
- 2025Rethinking Caching for LLM Serving Systems: Beyond Traditional Heuristics26 Aug 2025
- 2025HFX: Joint Design of Algorithms and Systems for Multi-SLO Serving and Fast Scaling21 Aug 2025
- 2025Efficient Serving of LLM Applications with Probabilistic Demand Modeling17 Jun 2025
- 2025Cost-Efficient LLM Serving in the Cloud: VM Selection with KV Cache Offloading16 Apr 2025
- 2025SageServe: Optimizing LLM Serving on Cloud Data Centers with Forecast Aware Auto-Scaling20 Feb 2025
- 2024A Survey on Large Language Models for Communication, Network, and Service Management: Application Insights, Challenges, and Future Directions16 Dec 2024
- 2024Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding20 Nov 2024
Similar, not cited
Close in wording, no citation either way
- 2025Optimal Scheduling Algorithms for LLM Inference: Theory and Practice
- 2026Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving
- 2026Taming Request Imbalance: SLO-Aware Scheduling for Disaggregated LLM Inference
- 2025AccelGen: Heterogeneous SLO-Guaranteed High-Throughput LLM Inference Serving for Diverse Applications
- 2025Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving
- 2025Scorpio: Serving Right Requests at the Right Time for Heterogeneous SLOs in LLM Inference
- 2025Memory Offloading for Large Language Model Inference with Latency SLO Guarantees
- 2024SCOOT: SLO-Oriented Performance Tuning for LLM Inference Engines