MaaSO: SLO-aware Orchestration of Heterogeneous Model Instances for MaaS
On the map: Efficient LLM inference and serving → Serving systems and request scheduling
Abstract (arXiv)
Model-as-a-Service (MaaS) platforms face diverse Service Level Objective (SLO) requirements stemming from various large language model (LLM) applications, manifested in contextual complexity, first-token latency, and between-token latency. On the other hand, an LLM instance, when configured with different parallelism strategies and inference batch sizes, exhibits distinct performance characteristics and can thus be used to serve different SLO requirements. However, current LLM inference systems typically deploy instances of the same model with identical configurations, lacking mechanisms to leverage such heterogeneity. To fill this research gap, we propose MaaSO, the first MaaS Orchestrator, which comprises three modules: (1) a profiler characterizing instance performance under diverse parallelism strategies and inference batch sizes; (2) a placer optimizing heterogeneous instance configurations; (3) a distributor enabling SLO-aware request distribution and preventing cascaded timeouts in continuous batching. Experiments show that MaaSO improves the SLO satisfaction ratio by 15 to 30% and reduces response latency by 40 to 60% compared to existing approaches, and significantly lowers overall orchestration overhead.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 17 of its 32 references that are in the map. First posted on arXiv 8 Sep 2025.
Builds on
17 of 32 references.
References · 17
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
- 2023Qwen Technical Reportcited by 3,273
- 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
- 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
- 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
- 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
- 2016Clipper: A Low-Latency Online Prediction Serving Systemcited by 191
- 2018Beyond Data and Model Parallelism for Deep Neural Networkscited by 177
- 2020Serving DNNs like Clockwork: Performance Predictability from the Bottom Upcited by 119
- 2022A Survey of Knowledge Enhanced Pre-trained Language Modelscited by 76
- 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
5 more
- 2023Model-as-a-Service (MaaS): A Surveycited by 24
- 2024Enhancing Document AI Data Generation Through Graph-Based Synthetic Layoutscited by 12
- 2024Control Industrial Automation System with Large Language Model Agentscited by 11
- 2025SLO-Aware Scheduling for Large Language Model Inferencescited by 7
- 2024Intelligent Condition Monitoring of Industrial Plants: An Overview of Methodologies and Uncertainty Management Strategiescited by 3
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025Scorpio: Serving Right Requests at the Right Time for Heterogeneous SLOs in LLM Inference
- 2026Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving
- 2026Taming Request Imbalance: SLO-Aware Scheduling for Disaggregated LLM Inference ↗
- 2025JITServe: SLO-aware LLM Serving with Imprecise Request Information
- 2025BrownoutServe: SLO-Aware Inference Serving under Bursty Workloads for MoE-based LLMs
- 2025Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving
- 2024Llumnix: Dynamic Scheduling for Large Language Model Serving
- 2026Beyond Binary Priorities: Multi-Tier SLA Scheduling for Large Language Model Serving