mlsanityData to 7 Oct 2026

Scorpio: Serving Right Requests at the Right Time for Heterogeneous SLOs in LLM Inference

arXiv 2505.23022 ↗First posted 29 May 2025cs.LGcited by 6

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Large Language Model (LLM) serving increasingly underpins online Web services such as conversational agents, Web search, and programming assistants, where requests carry heterogeneous Service Level Objectives (SLOs) such as Time to First Token (TTFT) and Time Per Output Token (TPOT). Existing LLM serving systems prioritize maximum throughput and treat all requests uniformly, which leads to suboptimal SLO attainment. This paper introduces Scorpio, an SLO-oriented LLM serving system designed to maximize system goodput and SLO attainment for workloads with heterogeneous SLOs. Our core insight is to exploit SLO heterogeneity for adaptive scheduling across admission control, queue management, and batch selection. Scorpio features a TTFT Guard, which employs least-deadline-first reordering and rejects unattainable requests, and a TPOT Guard, which utilizes a VBS-based admission control and a novel credit-based batching mechanism. Both guards are supported by a predictive module. Evaluations demonstrate that Scorpio improves system goodput by up to 14.4x and SLO adherence by up to 46.5% under high load compared to state-of-the-art baselines.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 23 of its 33 references that are in the map. First posted on arXiv 29 May 2025.

Builds on

23 of 33 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024FastSwitch: Optimizing Context Switching Efficiency in Fairness-aware Large Language Model Serving27 Nov
  2. 2024Efficient LLM Scheduling by Learning to Rank28 Aug
  3. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving24 Jun
  4. 2024Queue management for slo-oriented large language model serving5 Jun
  5. 2024Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction12 Apr
  6. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  7. 2023Fairness in Serving Large Language Models31 Dec
  8. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems23 Dec
  9. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills31 Aug

Other references · 14

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  4. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  6. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  7. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  8. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  9. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  10. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughputcited by 81
  11. 2024LLM Inference Serving: Survey of Recent Advances and Opportunitiescited by 46
  12. 2024Enabling Efficient Batch Serving for LMaaS via Generation Length Predictioncited by 20
2 more
  1. 2025SLOs-Serve: Optimized Serving of Multi-SLO LLMscited by 17
  2. 2025Hierarchical Prediction-based Management for LMaaS Systemscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 4.

6 papers.

  1. 2025HFX: Joint Design of Algorithms and Systems for Multi-SLO Serving and Fast Scalingcited by 2
  2. 2026LatencyPrism: Online Non-intrusive Latency Sculpting for SLO-Guaranteed LLM Inferencecited by 1
  3. 2025ProServe: Unified Multi-Priority Request Scheduling for LLM Servingcited by 0
  4. 2026Execution Envelopes: A Shared Admission Contract for Backend AI Execution Requestscited by 0
  5. 2026Beyond Binary Priorities: Multi-Tier SLA Scheduling for Large Language Model Servingcited by 0
  6. 2026FluidPD: In-Place Elasticity for SLO-Aware Prefill-Decode Disaggregated LLM Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving
  2. 2026Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving
  3. 2026Taming Request Imbalance: SLO-Aware Scheduling for Disaggregated LLM Inference ↗
  4. 2025JITServe: SLO-aware LLM Serving with Imprecise Request Information
  5. 2025Optimal Scheduling Algorithms for LLM Inference: Theory and Practice
  6. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving
  7. 2025Niyama : Breaking the Silos of LLM Inference Serving
  8. 2026Position: LLM Serving Needs Mathematical Optimization and Algorithmic Foundations, Not Just Heuristics