# Singularity: Planet-Scale, Preemptive and Elastic Scheduling of AI Workloads

arXiv: https://arxiv.org/abs/2202.07848 · HTML: https://mlsanity.com/paper/2202.07848/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 27 references that are in the map.
- First posted on arXiv 16 Feb 2022.

## Builds on (most cited first, 5 of 5)

- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learning](https://mlsanity.com/paper/2008.12260/) (2020)
- [Heterogeneity-Aware Cluster Scheduling Policies for Deep Learning Workloads](https://mlsanity.com/paper/2008.09213/) (2020)
- [Themis: Fair and Efficient GPU Cluster Scheduling](https://mlsanity.com/paper/1907.01484/) (2019)
- [MANA for MPI: MPI-Agnostic Network-Agnostic Transparent Checkpointing](https://mlsanity.com/paper/1904.12595/) (2019)

## Built on by (most cited first, 10 of 10)

- [Tenplex: Dynamic Parallelism for Deep Learning using Parallelizable Tensor Collections](https://mlsanity.com/paper/2312.05181/) (2023)
- [TRANSOM: An Efficient Fault-Tolerant System for Training LLMs](https://mlsanity.com/paper/2310.10046/) (2023)
- [POLCA: Power Oversubscription in LLM Cloud Providers](https://mlsanity.com/paper/2308.12908/) (2023)
- [Universal Checkpointing: A Flexible and Efficient Distributed Checkpointing System for Large-Scale DNN Training with Reconfigurable Parallelis](https://mlsanity.com/paper/2406.18820/) (2024)
- [Training LLMs with Fault Tolerant HSDP on 100,000 GPUs](https://mlsanity.com/paper/2602.00277/) (2026)
- [Maya: Optimizing Deep Learning Training Workloads using GPU Runtime Emulation](https://mlsanity.com/paper/2503.20191/) (2025)
- [Power-Flexible AI Data Centers: A New Paradigm for Grid-Responsive Compute](https://mlsanity.com/paper/2606.25098/) (2026)
- [Compass: A Decentralized Scheduler for Latency-Sensitive ML Workflows](https://mlsanity.com/paper/2402.17652/) (2024)
- [FlexNPU: Transparent NPU Virtualization for Dynamic LLM Prefill-Decode Co-location](https://mlsanity.com/paper/2606.04415/) (2026)
- [Efficient Resource-Shiftable RL Post-training with Generative Rewards](https://mlsanity.com/paper/2610.16866/) (2026)

## Abstract (arXiv)

Lowering costs by driving high utilization across deep learning workloads is a crucial lever for cloud providers. We present Singularity, Microsoft's globally distributed scheduling service for highly-efficient and reliable execution of deep learning training and inference workloads. At the heart of Singularity is a novel, workload-aware scheduler that can transparently preempt and elastically scale deep learning workloads to drive high utilization without impacting their correctness or performance, across a global fleet of AI accelerators (e.g., GPUs, FPGAs). All jobs in Singularity are preemptable, migratable, and dynamically resizable (elastic) by default: a live job can be dynamically and transparently (a) preempted and migrated to a different set of nodes, cluster, data center or a region and resumed exactly from the point where the execution was preempted, and (b) resized (i.e., elastically scaled-up/down) on a varying set of accelerators of a given type. Our mechanisms are transparent in that they do not require the user to make any changes to their code or require using any custom libraries that may limit flexibility. Additionally, our approach significantly improves the reliability of deep learning workloads. We show that the resulting efficiency and reliability gains with Singularity are achieved with negligible impact on the steady-state performance. Finally, our design approach is agnostic of DNN architectures and handles a variety of parallelism strategies (e.g., data/pipeline/model parallelism).