Global Simulation-Guided Dynamic Operator Scheduling for Efficient Multi-Tenant Model Serving
Not on the map. Why?
Abstract (arXiv)
Container-granularity scheduling leaves abundant short-lived idle slices within containers unexploited. Reallocating containers is too heavyweight to utilize such fine-grained opportunities under SLA constraints, and operator-level scheduling requires reasoning about dependencies, memory safety, and cluster-wide execution dynamics in real time. In this paper, we present SliceScheduler, a dynamic operator-level scheduling system for multi-tenant model serving. The key idea is to expose cluster-wide operator execution state and enable what-if reasoning over scheduling decisions. SliceScheduler consists of four key components. First, we introduce the Global Mapping Graph (GMG), a unified abstraction that captures operator dependencies, tensor shapes, resource mappings, and execution states, providing a real-time, cluster-wide view with explicit resource semantics. Second, we build a global simulator on top of GMG to predict operator-level execution and memory evolution under candidate placements. Third, we design an incremental, simulation-based scheduling module that selects placements to exploit fragmented idle slices while avoiding memory violations and preserving SLA. Finally, we develop an operator executor that materializes scheduling decisions on GPUs and coordinates computation and cross-accelerator transfers. We implement SliceScheduler as a PyTorch backend and evaluate it using production trace replay. Experimental results show that SliceScheduler improves token throughput by 1.10--2.29× compared to existing approaches, while maintaining SLA violations within 9%. SliceScheduler demonstrates that operator-level scheduling is a practical and effective approach to improving GPU utilization for multi-tenant LLM serving.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 13 of its 40 references that are in the map. First posted on arXiv 16 Aug 2026.
Builds on
13 of 40 references.
References · 13
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
- 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 527
- 2022Pathways: Asynchronous Distributed Dataflow for MLcited by 82
- 2019Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloadscited by 79
- 2020Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learningcited by 72
- 2023ASTRA-sim2.0: Modeling Hierarchical Networks and Disaggregated Systems for Large-model Training at Scalecited by 47
- 2024Characterization of Large Language Model Development in the Datacentercited by 46
- 2019Themis: Fair and Efficient GPU Cluster Schedulingcited by 43
- 2021Characterization and Prediction of Deep Learning Workloads in Large-Scale GPU Datacenterscited by 31
- 2022Aryl: An Elastic Cluster Scheduler for Deep Learningcited by 27
- 2022Shockwave: Fair and Efficient Cluster Scheduling for Dynamic Adaptation in Machine Learningcited by 12
- 2024HAP: SPMD DNN Training on Heterogeneous GPU Clusters with Automated Program Synthesiscited by 8
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026OpScale: Operator-level Provisioning and Autoscaling for LLM Serving
- 2026OpWeave: Flexible Operator Disaggregation for Heterogeneous LLM Serving
- 2026ScaleSim: Serving Large-Scale Multi-Agent Simulation with Invocation Distance-Based Memory Management
- 2024Improving Parallel Program Performance with LLM Optimizers via Agent-System Interfaces
- 2026Towards Multi-Model LLM Schedulers: Empirical Insights into Offloading and Preemption
- 2026OServe: Accelerating LLM Serving via Spatial-Temporal Workload Orchestration
- 2026MAPS: Memory-Aware Predictive Scheduling Framework for Large Language Model Serving
- 2026SAGA: Workflow-Atomic Scheduling for AI Agent Inference on GPU Clusters