# Global Simulation-Guided Dynamic Operator Scheduling for Efficient Multi-Tenant Model Serving

arXiv: https://arxiv.org/abs/2608.15762 · HTML: https://mlsanity.com/paper/2608.15762/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 40 references that are in the map.
- First posted on arXiv 16 Aug 2026.

## Builds on (most cited first, 13 of 13)

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Ray: A Distributed Framework for Emerging AI Applications](https://mlsanity.com/paper/1712.05889/) (2017)
- [Pathways: Asynchronous Distributed Dataflow for ML](https://mlsanity.com/paper/2203.12533/) (2022)
- [Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads](https://mlsanity.com/paper/1901.05758/) (2019)
- [Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learning](https://mlsanity.com/paper/2008.12260/) (2020)
- [ASTRA-sim2.0: Modeling Hierarchical Networks and Disaggregated Systems for Large-model Training at Scale](https://mlsanity.com/paper/2303.14006/) (2023)
- [Characterization of Large Language Model Development in the Datacenter](https://mlsanity.com/paper/2403.07648/) (2024)
- [Themis: Fair and Efficient GPU Cluster Scheduling](https://mlsanity.com/paper/1907.01484/) (2019)
- [Characterization and Prediction of Deep Learning Workloads in Large-Scale GPU Datacenters](https://mlsanity.com/paper/2109.01313/) (2021)
- [Aryl: An Elastic Cluster Scheduler for Deep Learning](https://mlsanity.com/paper/2202.07896/) (2022)
- [Shockwave: Fair and Efficient Cluster Scheduling for Dynamic Adaptation in Machine Learning](https://mlsanity.com/paper/2210.00093/) (2022)
- [HAP: SPMD DNN Training on Heterogeneous GPU Clusters with Automated Program Synthesis](https://mlsanity.com/paper/2401.05965/) (2024)
- [Online Evolutionary Batch Size Orchestration for Scheduling Deep Learning Workloads in GPU Clusters](https://mlsanity.com/paper/2108.03645/) (2021)

## Abstract (arXiv)

Container-granularity scheduling leaves abundant short-lived idle slices within containers unexploited. Reallocating containers is too heavyweight to utilize such fine-grained opportunities under SLA constraints, and operator-level scheduling requires reasoning about dependencies, memory safety, and cluster-wide execution dynamics in real time. In this paper, we present SliceScheduler, a dynamic operator-level scheduling system for multi-tenant model serving. The key idea is to expose cluster-wide operator execution state and enable what-if reasoning over scheduling decisions. SliceScheduler consists of four key components. First, we introduce the Global Mapping Graph (GMG), a unified abstraction that captures operator dependencies, tensor shapes, resource mappings, and execution states, providing a real-time, cluster-wide view with explicit resource semantics. Second, we build a global simulator on top of GMG to predict operator-level execution and memory evolution under candidate placements. Third, we design an incremental, simulation-based scheduling module that selects placements to exploit fragmented idle slices while avoiding memory violations and preserving SLA. Finally, we develop an operator executor that materializes scheduling decisions on GPUs and coordinates computation and cross-accelerator transfers. We implement SliceScheduler as a PyTorch backend and evaluate it using production trace replay. Experimental results show that SliceScheduler improves token throughput by 1.10--2.29× compared to existing approaches, while maintaining SLA violations within 9%. SliceScheduler demonstrates that operator-level scheduling is a practical and effective approach to improving GPU utilization for multi-tenant LLM serving.