# Effective Elastic Scaling of Deep Learning Workloads

arXiv: https://arxiv.org/abs/2006.13878 · HTML: https://mlsanity.com/paper/2006.13878/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Jun 2020.

## Built on by (most cited first, 4 of 4)

- [Deep Learning Workload Scheduling in GPU Datacenters: Taxonomy, Challenges and Vision](https://mlsanity.com/paper/2205.11913/) (2022)
- [End-to-end Adaptive Distributed Training on PaddlePaddle](https://mlsanity.com/paper/2112.02752/) (2021)
- [ANDREAS: Artificial intelligence traiNing scheDuler foR accElerAted resource clusterS](https://mlsanity.com/paper/2105.05080/) (2021)
- [Accelerating Transfer Learning with Near-Data Computation on Cloud Object Stores](https://mlsanity.com/paper/2210.08650/) (2022)

## Abstract (arXiv)

The increased use of deep learning (DL) in academia, government and industry has, in turn, led to the popularity of on-premise and cloud-hosted deep learning platforms, whose goals are to enable organizations utilize expensive resources effectively, and to share said resources among multiple teams in a fair and effective manner. In this paper, we examine the elastic scaling of Deep Learning (DL) jobs over large-scale training platforms and propose a novel resource allocation strategy for DL training jobs, resulting in improved job run time performance as well as increased cluster utilization. We begin by analyzing DL workloads and exploit the fact that DL jobs can be run with a range of batch sizes without affecting their final accuracy. We formulate an optimization problem that explores a dynamic batch size allocation to individual DL jobs based on their scaling efficiency, when running on multiple nodes. We design a fast dynamic programming based optimizer to solve this problem in real-time to determine jobs that can be scaled up/down, and use this optimizer in an autoscaler to dynamically change the allocated resources and batch sizes of individual DL jobs. We demonstrate empirically that our elastic scaling algorithm can complete up to ≈ 2 × as many jobs as compared to a strong baseline algorithm that also scales the number of GPUs but does not change the batch size. We also demonstrate that the average completion time with our algorithm is up to ≈ 10 × faster than that of the baseline.