# Heterogeneity-Aware Cluster Scheduling Policies for Deep Learning Workloads

arXiv: https://arxiv.org/abs/2008.09213 · HTML: https://mlsanity.com/paper/2008.09213/

## Key facts

- Cited by 56 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Aug 2020.

## Built on by (most cited first, 30 of 56)

- [Parrot: Efficient Serving of LLM-based Applications with Semantic Variable](https://mlsanity.com/paper/2405.19888/) (2024)
- [Pathways: Asynchronous Distributed Dataflow for ML](https://mlsanity.com/paper/2203.12533/) (2022)
- [Fairness in Serving Large Language Models](https://mlsanity.com/paper/2401.00588/) (2023)
- [Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNs](https://mlsanity.com/paper/2204.12013/) (2022)
- [Characterization and Prediction of Deep Learning Workloads in Large-Scale GPU Datacenters](https://mlsanity.com/paper/2109.01313/) (2021)
- [M'elange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneity](https://mlsanity.com/paper/2404.14527/) (2024)
- [A Runtime-Based Computational Performance Predictor for Deep Neural Network Training](https://mlsanity.com/paper/2102.00527/) (2021)
- [Aryl: An Elastic Cluster Scheduler for Deep Learning](https://mlsanity.com/paper/2202.07896/) (2022)
- [vTrain: A Simulation Framework for Evaluating Cost-effective and Compute-optimal Large Language Model Training](https://mlsanity.com/paper/2312.12391/) (2023)
- [KAISA: An Adaptive Second-Order Optimizer Framework for Deep Neural Networks](https://mlsanity.com/paper/2107.01739/) (2021)
- [HeterPS: Distributed Deep Learning With Reinforcement Learning Based Scheduling in Heterogeneous Environments](https://mlsanity.com/paper/2111.10635/) (2021)
- [Singularity: Planet-Scale, Preemptive and Elastic Scheduling of AI Workloads](https://mlsanity.com/paper/2202.07848/) (2022)
- [Communication-Efficient Large-Scale Distributed Deep Learning: A Comprehensive Survey](https://mlsanity.com/paper/2404.06114/) (2024)
- [GRANITE: A Graph Neural Network Model for Basic Block Throughput Estimation](https://mlsanity.com/paper/2210.03894/) (2022)
- [ELIS: Efficient LLM Iterative Scheduling System with Response Length Predictor](https://mlsanity.com/paper/2505.09142/) (2025)
- [Serving DNN Models with Multi-Instance GPUs: A Case of the Reconfigurable Machine Scheduling Problem](https://mlsanity.com/paper/2109.11067/) (2021)
- [Spindle: Efficient Distributed Training of Multi-Task Large Models via Wavefront Scheduling](https://mlsanity.com/paper/2409.03365/) (2024)
- [Optimizing Mixture-of-Experts Inference Time Combining Model Deployment and Communication Scheduling](https://mlsanity.com/paper/2410.17043/) (2024)
- [Deep Learning Workload Scheduling in GPU Datacenters: Taxonomy, Challenges and Vision](https://mlsanity.com/paper/2205.11913/) (2022)
- [PLoRA: Efficient Concurrent LoRA Training for Large Language Models](https://mlsanity.com/paper/2508.02932/) (2025)
- [Batch Query Processing and Optimization for Agentic Workflows](https://mlsanity.com/paper/2509.02121/) (2025)
- [Shared Memory-contention-aware Concurrent DNN Execution for Diversely Heterogeneous System-on-Chips](https://mlsanity.com/paper/2308.05869/) (2023)
- [Taming the Chaos: Coordinated Autoscaling for Heterogeneous and Disaggregated LLM Inference](https://mlsanity.com/paper/2508.19559/) (2025)
- [KAIROS: Building Cost-Efficient Machine Learning Inference Systems with Heterogeneous Cloud Resources](https://mlsanity.com/paper/2210.05889/) (2022)
- [InTune: Reinforcement Learning-based Data Pipeline Optimization for Deep Recommendation Models](https://mlsanity.com/paper/2308.08500/) (2023)
- [Ensuring Fair LLM Serving Amid Diverse Applications](https://mlsanity.com/paper/2411.15997/) (2024)
- [Saturn: An Optimized Data System for Large Model Deep Learning Workloads](https://mlsanity.com/paper/2309.01226/) (2023)
- [Computing in the Era of Large Generative Models: From Cloud-Native to AI-Native](https://mlsanity.com/paper/2401.12230/) (2024)
- [SmartGen: Seamless Disaggregated LLM Inference with Selective KV Cache Transfer](https://mlsanity.com/paper/2607.28150/) (2026)
- [TENSILE: A Tensor granularity dynamic GPU memory scheduling method toward multiple dynamic workloads system](https://mlsanity.com/paper/2105.13336/) (2021)

## Abstract (arXiv)

Specialized accelerators such as GPUs, TPUs, FPGAs, and custom ASICs have been increasingly deployed to train deep learning models. These accelerators exhibit heterogeneous performance behavior across model architectures. Existing schedulers for clusters of accelerators, which are used to arbitrate these expensive training resources across many users, have shown how to optimize for various multi-job, multi-user objectives, like fairness and makespan. Unfortunately, existing schedulers largely do not consider performance heterogeneity. In this paper, we propose Gavel, a heterogeneity-aware scheduler that systematically generalizes a wide range of existing scheduling policies. Gavel expresses these policies as optimization problems, making it easy to optimize for objectives in a heterogeneity-aware way, while also being cognizant of performance optimizations like space sharing. Gavel then uses a round-based scheduling mechanism to ensure jobs receive their ideal allocation given the target scheduling policy. Gavel's heterogeneity-aware policies allow a heterogeneous cluster to sustain higher input load, and improve end objectives such as average job completion time and makespan by up to 3.5x compared to heterogeneity-agnostic policies.