mlsanityData to 14 Oct 2026

Taming Resource Heterogeneity In Distributed ML Training With Dynamic Batching

arXiv 2305.12213 ↗First posted 20 May 2023cs.LG cs.DCcited by 12

Not on the map. Why?

Abstract (arXiv)

Current techniques and systems for distributed model training mostly assume that clusters are comprised of homogeneous servers with a constant resource availability. However, cluster heterogeneity is pervasive in computing infrastructure, and is a fundamental characteristic of low-cost transient resources (such as EC2 spot instances). In this paper, we develop a dynamic batching technique for distributed data-parallel training that adjusts the mini-batch sizes on each worker based on its resource availability and throughput. Our mini-batch controller seeks to equalize iteration times on all workers, and facilitates training on clusters comprised of servers with different amounts of CPU and GPU resources. This variable mini-batch technique uses proportional control and ideas from PID controllers to find stable mini-batch sizes. Our empirical evaluation shows that dynamic batching can reduce model training times by more than 4x on heterogeneous clusters.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 20 of its 43 references that are in the map. First posted on arXiv 20 May 2023.

Builds on

20 of 43 references.

References · 20

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2016Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learningcited by 1,512
  4. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 320
  5. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 193
  6. 2018Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysiscited by 156
  7. 2017Revisiting Distributed Synchronous SGDcited by 142
  8. 2017A Bayesian Perspective on Generalization and Stochastic Gradient Descentcited by 135
  9. 2019Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronizationcited by 107
  10. 2018Slow and Stale Gradients Can Win the Race: Error-Runtime Trade-offs in Distributed SGDcited by 68
  11. 2019Scalable Deep Learning on Distributed Infrastructures: Challenges, Techniques and Toolscited by 40
  12. 2017AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networkscited by 29
8 more
  1. 2015Model Accuracy and Runtime Tradeoff in Distributed Deep Learning:A Systematic Studycited by 19
  2. 2019On the Computation and Communication Complexity of Parallel SGD with Dynamic Batch Sizes for Stochastic Non-Convex Optimizationcited by 15
  3. 2019Hop: Heterogeneity-Aware Decentralized Trainingcited by 10
  4. 2020Anytime MiniBatch: Exploiting Stragglers in Online Distributed Optimizationcited by 9
  5. 2020Characterizing and Modeling Distributed Training with Transient Cloud GPU Serverscited by 8
  6. 2019Heterogeneity-aware Gradient Coding for Straggler Tolerancecited by 6
  7. 2017Portfolio-driven Resource Management for Transient Cloud Serverscited by 4
  8. 2019Speeding up Deep Learning with Transient Serverscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 2; 2025: 3; 2026: 2.

12 papers.

  1. 2023DYNAMITE: Dynamic Interplay of Mini-Batch Size and Aggregation Frequency for Federated Learning with Static and Streaming Datasetcited by 14
  2. 2023MergeSFL: Split Federated Learning with Feature Merging and Batch Size Regulationcited by 13
  3. 2023Accelerating Distributed ML Training via Selective Synchronizationcited by 7
  4. 2025OmniLearn: A Framework for Distributed Deep Learning over Heterogeneous Clusterscited by 5
  5. 2024Sustainable self-supervised learning for speech representationscited by 4
  6. 2023Efficiency-oriented approaches for self-supervised speech representation learningcited by 1
  7. 2024AntDT: A Self-Adaptive Distributed Training Framework for Leader and Straggler Nodescited by 1
  8. 2025OmniFed: A Modular Framework for Configurable Federated Learning from Edge to HPCcited by 1
  9. 2023ABS-SGD: A Delayed Synchronous Stochastic Gradient Descent Algorithm with Adaptive Batch Size for Heterogeneous GPU Clusterscited by 0
  10. 2025On Using Large-Batches in Federated Learningcited by 0
  11. 2026Tula: Optimizing Time, Cost, and Generalization in Distributed Large-Batch Trainingcited by 0
  12. 2026A Readiness-Driven Runtime for Pipeline-Parallel Training under Runtime Variabilitycited by 0