# SLO-Aware Scheduling for Large Language Model Inferences

arXiv: https://arxiv.org/abs/2504.14966 · HTML: https://mlsanity.com/paper/2504.14966/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Apr 2025.

## Built on by (most cited first, 7 of 7)

- [HFX: Joint Design of Algorithms and Systems for Multi-SLO Serving and Fast Scaling](https://mlsanity.com/paper/2508.15919/) (2025)
- [SuperInfer: SLO-Aware Rotary Scheduling and Memory Management for LLM Inference on Superchips](https://mlsanity.com/paper/2601.20309/) (2026)
- [MaaSO: SLO-aware Orchestration of Heterogeneous Model Instances for MaaS](https://mlsanity.com/paper/2509.06362/) (2025)
- [Mind the Delay: Latency-aware Query Routing for Heterogeneous LLM pools](https://mlsanity.com/paper/2607.18253/) (2026)
- [A Predictive and Synergistic Two-Layer Scheduling Framework for LLM Serving](https://mlsanity.com/paper/2509.23384/) (2025)
- [ProServe: Unified Multi-Priority Request Scheduling for LLM Serving](https://mlsanity.com/paper/2512.12928/) (2025)
- [TCM-Serve: Modality-aware Scheduling for Multimodal Large Language Model Inference](https://mlsanity.com/paper/2603.26498/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have revolutionized applications such as code completion, chatbots, and online classification. To elevate user experiences, service level objectives (SLOs) serve as crucial benchmarks for assessing inference services capabilities. In practice, an inference service processes multiple types of tasks, each with its own distinct SLO. To ensure satisfactory user experiences, each request's distinct SLOs should be considered in scheduling. However, existing designs lack this consideration, leading to insufficient hardware utility and suboptimal performance. This paper analyzes scenarios to process tasks with varying SLOs, and introduces a simulated annealing-based scheduler to decide request priority sequence based on a request's SLO, input lengths, and possible output lengths. As the first specialized scheduler for multi-SLO scenarios, this work improves SLO attainment by up to 5x and reduces average latency by 31.6% on Python-Code-23k-ShareGPT and ShareGPT_Vicuna_unfiltered datasets, compared to current state-of-the-art framework vLLM and a new framework LMDeploy.