mlsanityData to 7 Oct 2026

Online Evolutionary Batch Size Orchestration for Scheduling Deep Learning Workloads in GPU Clusters

arXiv 2108.03645 ↗First posted 8 Aug 2021cs.DC cs.AI cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

Efficient GPU resource scheduling is essential to maximize resource utilization and save training costs for the increasing amount of deep learning workloads in shared GPU clusters. Existing GPU schedulers largely rely on static policies to leverage the performance characteristics of deep learning jobs. However, they can hardly reach optimal efficiency due to the lack of elasticity. To address the problem, we propose ONES, an ONline Evolutionary Scheduler for elastic batch size orchestration. ONES automatically manages the elasticity of each job based on the training batch size, so as to maximize GPU utilization and improve scheduling efficiency. It determines the batch size for each job through an online evolutionary search that can continuously optimize the scheduling decisions. We evaluate the effectiveness of ONES with 64 GPUs on TACC's Longhorn supercomputers. The results show that ONES can outperform the prior deep learning schedulers with a significantly shorter average job completion time.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 24 of its 63 references that are in the map. First posted on arXiv 8 Aug 2021.

Builds on

24 of 63 references.

References · 24

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  5. 2014Going Deeper with Convolutionscited by 4,231
  6. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  7. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  8. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  9. 2015Cyclical Learning Rates for Training Neural Networkscited by 627
  10. 2017Large Batch Training of Convolutional Networkscited by 529
  11. 2015Deep Speech 2: End-to-End Speech Recognition in English and Mandarincited by 449
  12. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 313
12 more
  1. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 278
  2. 2018Learning Scheduling Algorithms for Data Processing Clusterscited by 133
  3. 2019Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloadscited by 79
  4. 2021Go Wider Instead of Deepercited by 56
  5. 2019Large-Batch Training for LSTM and Beyondcited by 21
  6. 2019DL2: A Deep Learning-driven Scheduler for Deep Learning Clusterscited by 14
  7. 2021Concurrent Adversarial Learning for Large-Batch Trainingcited by 11
  8. 2018SLAQ: Quality-Driven Scheduling for Distributed Machine Learningcited by 10
  9. 2018Online Job Scheduling in Distributed Machine Learning Clusterscited by 9
  10. 2019Dynamic Mini-batch SGD for Elastic Distributed Training: Learning in the Limbo of Resourcescited by 7
  11. 2019Auto-Precision Scaling for Distributed Deep Learningcited by 3
  12. 2020Training EfficientNets at Supercomputer Scale: 83% ImageNet Top-1 Accuracy in One Hourcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 2; 2024: 1; 2025: 0; 2026: 2.

7 papers.

  1. 2023vTrain: A Simulation Framework for Evaluating Cost-effective and Compute-optimal Large Language Model Trainingcited by 22
  2. 2022Deep Learning Workload Scheduling in GPU Datacenters: Taxonomy, Challenges and Visioncited by 6
  3. 2024GPU Cluster Scheduling for Network-Sensitive Deep Learningcited by 2
  4. 2022Hyper-Learning for Gradient-Based Batch Size Adaptationcited by 0
  5. 2023NEPHELE: A Neural Platform for Highly Realistic Cloud Radiance Renderingcited by 0
  6. 2026What Artificial Intelligence can do for High-Performance Computing systems?cited by 0
  7. 2026Global Simulation-Guided Dynamic Operator Scheduling for Efficient Multi-Tenant Model Servingcited by 0