# Aryl: An Elastic Cluster Scheduler for Deep Learning

arXiv: https://arxiv.org/abs/2202.07896 · HTML: https://mlsanity.com/paper/2202.07896/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 72 references that are in the map.
- First posted on arXiv 16 Feb 2022.

## Builds on (most cited first, 19 of 19)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Large Batch Optimization for Deep Learning: Training BERT in 76 minutes](https://mlsanity.com/paper/1904.00962/) (2019)
- [Clipper: A Low-Latency Online Prediction Serving System](https://mlsanity.com/paper/1612.03079/) (2016)
- [Serving DNNs like Clockwork: Performance Predictability from the Bottom Up](https://mlsanity.com/paper/2006.02464/) (2020)
- [MLPerf Training Benchmark](https://mlsanity.com/paper/1910.01500/) (2019)
- [Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learning](https://mlsanity.com/paper/2008.12260/) (2020)
- [Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes](https://mlsanity.com/paper/1807.11205/) (2018)
- [Heterogeneity-Aware Cluster Scheduling Policies for Deep Learning Workloads](https://mlsanity.com/paper/2008.09213/) (2020)
- [HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through Integration of Pipelined Model Parallelism and Data Parallelism](https://mlsanity.com/paper/2005.14038/) (2020)
- [Deep Learning Inference in Facebook Data Centers: Characterization, Performance Optimizations and Hardware Implications](https://mlsanity.com/paper/1811.09886/) (2018)
- [A Runtime-Based Computational Performance Predictor for Deep Neural Network Training](https://mlsanity.com/paper/2102.00527/) (2021)
- [Daydream: Accurately Estimating the Efficacy of Optimizations for DNN Training](https://mlsanity.com/paper/2006.03318/) (2020)
- [Characterizing Concurrency Mechanisms for NVIDIA GPUs under Deep Learning Workloads](https://mlsanity.com/paper/2110.00459/) (2021)

## Built on by (most cited first, 27 of 27)

- [Characterization of Large Language Model Development in the Datacenter](https://mlsanity.com/paper/2403.07648/) (2024)
- [ByteCheckpoint: A Unified Checkpointing System for Large Foundation Model Development](https://mlsanity.com/paper/2407.20143/) (2024)
- [Tenplex: Dynamic Parallelism for Deep Learning using Parallelizable Tensor Collections](https://mlsanity.com/paper/2312.05181/) (2023)
- [Unicron: Economizing Self-Healing LLM Training at Scale](https://mlsanity.com/paper/2401.00134/) (2023)
- [TRANSOM: An Efficient Fault-Tolerant System for Training LLMs](https://mlsanity.com/paper/2310.10046/) (2023)
- [Communication-Efficient Large-Scale Distributed Deep Learning: A Comprehensive Survey](https://mlsanity.com/paper/2404.06114/) (2024)
- [FlexLLM: Token-Level Co-Serving of LLM Inference and Finetuning with SLO Guarantees](https://mlsanity.com/paper/2402.18789/) (2024)
- [Lazarus: Resilient and Elastic Training of Mixture-of-Experts Models](https://mlsanity.com/paper/2407.04656/) (2024)
- [ELIS: Efficient LLM Iterative Scheduling System with Response Length Predictor](https://mlsanity.com/paper/2505.09142/) (2025)
- [Spindle: Efficient Distributed Training of Multi-Task Large Models via Wavefront Scheduling](https://mlsanity.com/paper/2409.03365/) (2024)
- [InfiniteHBD: Building Datacenter-Scale High-Bandwidth Domain for LLM with Optical Circuit Switching Transceivers](https://mlsanity.com/paper/2502.03885/) (2025)
- [Libra: Efficient Resource Management for Agentic RL Post-Training](https://mlsanity.com/paper/2606.03077/) (2026)
- [LeMix: Unified Scheduling for LLM Training and Inference on Multi-GPU Systems](https://mlsanity.com/paper/2507.21276/) (2025)
- [PipeLive: Efficient Live In-place Pipeline Parallelism Reconfiguration for Dynamic LLM Serving](https://mlsanity.com/paper/2604.12171/) (2026)
- [Resource Allocation and Workload Scheduling for Large-Scale Distributed Deep Learning: A Survey](https://mlsanity.com/paper/2406.08115/) (2024)
- [MACE: A Hybrid LLM Serving System with Colocated SLO-aware Continuous Retraining Alignment](https://mlsanity.com/paper/2510.03283/) (2025)
- [xMem: A CPU-Based Approach for Accurate Estimation of GPU Memory in Deep Learning Training Workloads](https://mlsanity.com/paper/2510.21048/) (2025)
- [EDiT: A Local-SGD-Based Efficient Distributed Training Method for Large Language Models](https://mlsanity.com/paper/2412.07210/) (2024)
- [SpecInF: Exploiting Idle GPU Resources in Distributed DL Training via Speculative Inference Filling](https://mlsanity.com/paper/2503.02550/) (2025)
- [Tesserae: Scalable Placement Policies for Deep Learning Workloads](https://mlsanity.com/paper/2508.04953/) (2025)
- [AEGIS: Runtime-Guided GPU Collocation for Multi-Tenant Deep Learning Training](https://mlsanity.com/paper/2508.19073/) (2025)
- [Smoothed Online Optimization for Target Tracking: Robust and Learning-Augmented Algorithms](https://mlsanity.com/paper/2509.05930/) (2025)
- [Hybrid Learning and Optimization-Based Dynamic Scheduling for DL Workloads on Heterogeneous GPU Clusters](https://mlsanity.com/paper/2512.10271/) (2025)
- [CoLLM: Continuous Adaptation for SLO-Aware LLM Serving on Shared GPU Clusters](https://mlsanity.com/paper/2604.16400/) (2026)
- [DeltaServe: Host-Agnostic Co-Serving of Inference and Fine-Tuning for LLMs](https://mlsanity.com/paper/2607.28848/) (2026)
- [Global Simulation-Guided Dynamic Operator Scheduling for Efficient Multi-Tenant Model Serving](https://mlsanity.com/paper/2608.15762/) (2026)
- [COMPASS-ABS: Reducing Fragmentation in Shared GPU Clusters for Deep Learning Training Workloads](https://mlsanity.com/paper/2609.18519/) (2026)

## Abstract (arXiv)

Companies build separate training and inference GPU clusters for deep learning, and use separate schedulers to manage them. This leads to problems for both training and inference: inference clusters have low GPU utilization when the traffic load is low; training jobs often experience long queueing time due to lack of resources. We introduce Aryl, a new cluster scheduler to address these problems. Aryl introduces capacity loaning to loan idle inference GPU servers for training jobs. It further exploits elastic scaling that scales a training job's GPU allocation to better utilize loaned resources. Capacity loaning and elastic scaling create new challenges to cluster management. When the loaned servers need to be returned, we need to minimize the number of job preemptions; when more GPUs become available, we need to allocate them to elastic jobs and minimize the job completion time (JCT). Aryl addresses these combinatorial problems using principled heuristics. It introduces the notion of server preemption cost which it greedily reduces during server reclaiming. It further relies on the JCT reduction value defined for each additional worker for an elastic job to solve the scheduling problem as a multiple-choice knapsack problem. Prototype implementation on a 64-GPU testbed and large-scale simulation with 15-day traces of over 50,000 production jobs show that Aryl brings 1.53x and 1.50x reductions in average queuing time and JCT, and improves cluster usage by up to 26.9% over the cluster scheduler without capacity loaning or elastic scaling.