# Online Evolutionary Batch Size Orchestration for Scheduling Deep Learning Workloads in GPU Clusters

arXiv: https://arxiv.org/abs/2108.03645 · HTML: https://mlsanity.com/paper/2108.03645/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 63 references that are in the map.
- First posted on arXiv 8 Aug 2021.

## Builds on (most cited first, 24 of 24)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [Large Batch Optimization for Deep Learning: Training BERT in 76 minutes](https://mlsanity.com/paper/1904.00962/) (2019)
- [Cyclical Learning Rates for Training Neural Networks](https://mlsanity.com/paper/1506.01186/) (2015)
- [Large Batch Training of Convolutional Networks](https://mlsanity.com/paper/1708.03888/) (2017)
- [Deep Speech 2: End-to-End Speech Recognition in English and Mandarin](https://mlsanity.com/paper/1512.02595/) (2015)
- [Don't Decay the Learning Rate, Increase the Batch Size](https://mlsanity.com/paper/1711.00489/) (2017)
- [Train longer, generalize better: closing the generalization gap in large batch training of neural networks](https://mlsanity.com/paper/1705.08741/) (2017)
- [Learning Scheduling Algorithms for Data Processing Clusters](https://mlsanity.com/paper/1810.01963/) (2018)
- [Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads](https://mlsanity.com/paper/1901.05758/) (2019)
- [Go Wider Instead of Deeper](https://mlsanity.com/paper/2107.11817/) (2021)
- [Large-Batch Training for LSTM and Beyond](https://mlsanity.com/paper/1901.08256/) (2019)
- [DL2: A Deep Learning-driven Scheduler for Deep Learning Clusters](https://mlsanity.com/paper/1909.06040/) (2019)
- [Concurrent Adversarial Learning for Large-Batch Training](https://mlsanity.com/paper/2106.00221/) (2021)
- [SLAQ: Quality-Driven Scheduling for Distributed Machine Learning](https://mlsanity.com/paper/1802.04819/) (2018)
- [Online Job Scheduling in Distributed Machine Learning Clusters](https://mlsanity.com/paper/1801.00936/) (2018)
- [Dynamic Mini-batch SGD for Elastic Distributed Training: Learning in the Limbo of Resources](https://mlsanity.com/paper/1904.12043/) (2019)
- [Auto-Precision Scaling for Distributed Deep Learning](https://mlsanity.com/paper/1911.08907/) (2019)
- [Training EfficientNets at Supercomputer Scale: 83% ImageNet Top-1 Accuracy in One Hour](https://mlsanity.com/paper/2011.00071/) (2020)

## Built on by (most cited first, 7 of 7)

- [vTrain: A Simulation Framework for Evaluating Cost-effective and Compute-optimal Large Language Model Training](https://mlsanity.com/paper/2312.12391/) (2023)
- [Deep Learning Workload Scheduling in GPU Datacenters: Taxonomy, Challenges and Vision](https://mlsanity.com/paper/2205.11913/) (2022)
- [GPU Cluster Scheduling for Network-Sensitive Deep Learning](https://mlsanity.com/paper/2401.16492/) (2024)
- [Hyper-Learning for Gradient-Based Batch Size Adaptation](https://mlsanity.com/paper/2205.08231/) (2022)
- [NEPHELE: A Neural Platform for Highly Realistic Cloud Radiance Rendering](https://mlsanity.com/paper/2303.04086/) (2023)
- [What Artificial Intelligence can do for High-Performance Computing systems?](https://mlsanity.com/paper/2602.00014/) (2026)
- [Global Simulation-Guided Dynamic Operator Scheduling for Efficient Multi-Tenant Model Serving](https://mlsanity.com/paper/2608.15762/) (2026)

## Abstract (arXiv)

Efficient GPU resource scheduling is essential to maximize resource utilization and save training costs for the increasing amount of deep learning workloads in shared GPU clusters. Existing GPU schedulers largely rely on static policies to leverage the performance characteristics of deep learning jobs. However, they can hardly reach optimal efficiency due to the lack of elasticity. To address the problem, we propose ONES, an ONline Evolutionary Scheduler for elastic batch size orchestration. ONES automatically manages the elasticity of each job based on the training batch size, so as to maximize GPU utilization and improve scheduling efficiency. It determines the batch size for each job through an online evolutionary search that can continuously optimize the scheduling decisions. We evaluate the effectiveness of ONES with 64 GPUs on TACC's Longhorn supercomputers. The results show that ONES can outperform the prior deep learning schedulers with a significantly shorter average job completion time.