mlsanityData to 7 Oct 2026

SLOs-Serve: Optimized Serving of Multi-SLO LLMs

arXiv 2504.08784 ↗First posted 5 Apr 2025cs.DC cs.LGcited by 17

Not on the map. Why?

Abstract (arXiv)

This paper introduces SLOs-Serve, a system designed for serving multi-stage large language model (LLM) requests with application- and stage-specific service level objectives (SLOs). The key idea behind SLOs-Serve is to customize the allocation of tokens to meet these SLO requirements. SLOs-Serve uses a multi-SLO dynamic programming-based algorithm to continuously optimize token allocations under SLO constraints by exploring the full design space of chunked prefill and (optional) speculative decoding. Leveraging this resource planning algorithm, SLOs-Serve effectively supports multi-SLOs and multi-replica serving with dynamic request routing while being resilient to bursty arrivals. Our evaluation across 6 LLM application scenarios (including summarization, coding, chatbot, tool calling, and reasoning) demonstrates that SLOs-Serve improves per-GPU serving capacity by 2.2x on average compared to prior state-of-the-art systems.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 24 of its 38 references that are in the map. First posted on arXiv 5 Apr 2025.

Builds on

24 of 38 references.

References · 24

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  4. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  5. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  6. 2025s1: Simple test-time scalingcited by 1,311
  7. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  8. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  9. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  10. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 516
  11. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  12. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 477
12 more
  1. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 210
  2. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 193
  3. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 120
  4. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelismcited by 110
  5. 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttentioncited by 93
  6. 2023Fairness in Serving Large Language Modelscited by 77
  7. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUscited by 73
  8. 2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Poolcited by 67
  9. 2024Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flowcited by 54
  10. 2024NeuPIMs: NPU-PIM Heterogeneous Acceleration for Batched LLM Inferencingcited by 48
  11. 2024ExeGPT: Constraint-Aware Resource Scheduling for LLM Inferencecited by 43
  12. 2023Apparate: Rethinking Early Exits to Tame Latency-Throughput Tensions in ML Servingcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 10.

17 papers.

Sort:

By area:

  1. 2025Prism: Cost-Efficient Multi-LLM Serving via GPU Memory Ballooningcited by 23
  2. 2025TokenFlow: Responsive LLM Text Streaming Serving under Request Burst via Preemptive Schedulingcited by 10
  3. 2025TokenScale: Timely and Accurate Autoscaling for Disaggregated LLM Serving with Token Velocitycited by 9
  4. 2025Scorpio: Serving Right Requests at the Right Time for Heterogeneous SLOs in LLM Inferencecited by 6
  5. 2025Astrolabe: Balancing Load in LLM Serving with Randomized Prediction-Guided Schedulingcited by 2
  6. 2026Simple is Better: Multiplication May Be All You Need for LLM Request Schedulingcited by 2
  7. 2026SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Schedulingcited by 2
  8. 2026PowerSlider: Exploiting Phase Asymmetry for LLM Serving under Demand Responsecited by 2
  9. 2026Stage-Aware Communication Scheduling for Disaggregated LLM Servingcited by 1
  10. 2025A Predictive and Synergistic Two-Layer Scheduling Framework for LLM Servingcited by 0
  11. 2025ProServe: Unified Multi-Priority Request Scheduling for LLM Servingcited by 0
  12. 2026Dooly: Configuration-Agnostic, Redundancy-Aware Profiling for LLM Inference Simulationcited by 0
5 more
  1. 2026Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference servingcited by 0
  2. 2026Beyond Binary Priorities: Multi-Tier SLA Scheduling for Large Language Model Servingcited by 0
  3. 2026FleetSieve: Decision-Critical Profiling for SLO-Aware LLM Fleet Configurationcited by 0
  4. 2026OpWeave: Flexible Operator Disaggregation for Heterogeneous LLM Servingcited by 0
  5. 2026DPS: Dual-Mode Precision LLM Serving with Semi-Unified Memorycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025AdaServe: Accelerating Multi-SLO LLM Serving with SLO-Customized Speculative Decoding
  2. 2024Queue management for slo-oriented large language model serving
  3. 2025JITServe: SLO-aware LLM Serving with Imprecise Request Information
  4. 2024Revisiting Service Level Objectives and System Level Metrics in Large Language Model Serving
  5. 2025HFX: Joint Design of Algorithms and Systems for Multi-SLO Serving and Fast Scaling
  6. 2025Ascendra: Dynamic Request Prioritization for Efficient LLM Serving
  7. 2025Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving
  8. 2026OrbitFlow: SLO-Aware Long-Context LLM Serving with Fine-Grained KV Cache Reconfiguration