# Taming Resource Heterogeneity In Distributed ML Training With Dynamic Batching

arXiv: https://arxiv.org/abs/2305.12213 · HTML: https://mlsanity.com/paper/2305.12213/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 43 references that are in the map.
- First posted on arXiv 20 May 2023.

## Builds on (most cited first, 20 of 20)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [Don't Decay the Learning Rate, Increase the Batch Size](https://mlsanity.com/paper/1711.00489/) (2017)
- [Measuring the Effects of Data Parallelism on Neural Network Training](https://mlsanity.com/paper/1811.03600/) (2018)
- [Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis](https://mlsanity.com/paper/1802.09941/) (2018)
- [Revisiting Distributed Synchronous SGD](https://mlsanity.com/paper/1702.05800/) (2017)
- [A Bayesian Perspective on Generalization and Stochastic Gradient Descent](https://mlsanity.com/paper/1710.06451/) (2017)
- [Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronization](https://mlsanity.com/paper/1910.13598/) (2019)
- [Slow and Stale Gradients Can Win the Race: Error-Runtime Trade-offs in Distributed SGD](https://mlsanity.com/paper/1803.01113/) (2018)
- [Scalable Deep Learning on Distributed Infrastructures: Challenges, Techniques and Tools](https://mlsanity.com/paper/1903.11314/) (2019)
- [AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networks](https://mlsanity.com/paper/1712.02029/) (2017)
- [Model Accuracy and Runtime Tradeoff in Distributed Deep Learning:A Systematic Study](https://mlsanity.com/paper/1509.04210/) (2015)
- [On the Computation and Communication Complexity of Parallel SGD with Dynamic Batch Sizes for Stochastic Non-Convex Optimization](https://mlsanity.com/paper/1905.04346/) (2019)
- [Hop: Heterogeneity-Aware Decentralized Training](https://mlsanity.com/paper/1902.01064/) (2019)
- [Anytime MiniBatch: Exploiting Stragglers in Online Distributed Optimization](https://mlsanity.com/paper/2006.05752/) (2020)
- [Characterizing and Modeling Distributed Training with Transient Cloud GPU Servers](https://mlsanity.com/paper/2004.03072/) (2020)
- [Heterogeneity-aware Gradient Coding for Straggler Tolerance](https://mlsanity.com/paper/1901.09339/) (2019)
- [Portfolio-driven Resource Management for Transient Cloud Servers](https://mlsanity.com/paper/1704.08738/) (2017)
- [Speeding up Deep Learning with Transient Servers](https://mlsanity.com/paper/1903.00045/) (2019)

## Built on by (most cited first, 12 of 12)

- [DYNAMITE: Dynamic Interplay of Mini-Batch Size and Aggregation Frequency for Federated Learning with Static and Streaming Dataset](https://mlsanity.com/paper/2310.14906/) (2023)
- [MergeSFL: Split Federated Learning with Feature Merging and Batch Size Regulation](https://mlsanity.com/paper/2311.13348/) (2023)
- [Accelerating Distributed ML Training via Selective Synchronization](https://mlsanity.com/paper/2307.07950/) (2023)
- [OmniLearn: A Framework for Distributed Deep Learning over Heterogeneous Clusters](https://mlsanity.com/paper/2503.17469/) (2025)
- [Sustainable self-supervised learning for speech representations](https://mlsanity.com/paper/2406.07696/) (2024)
- [Efficiency-oriented approaches for self-supervised speech representation learning](https://mlsanity.com/paper/2312.11142/) (2023)
- [AntDT: A Self-Adaptive Distributed Training Framework for Leader and Straggler Nodes](https://mlsanity.com/paper/2404.09679/) (2024)
- [OmniFed: A Modular Framework for Configurable Federated Learning from Edge to HPC](https://mlsanity.com/paper/2509.19396/) (2025)
- [ABS-SGD: A Delayed Synchronous Stochastic Gradient Descent Algorithm with Adaptive Batch Size for Heterogeneous GPU Clusters](https://mlsanity.com/paper/2308.15164/) (2023)
- [On Using Large-Batches in Federated Learning](https://mlsanity.com/paper/2509.10537/) (2025)
- [Tula: Optimizing Time, Cost, and Generalization in Distributed Large-Batch Training](https://mlsanity.com/paper/2603.18112/) (2026)
- [A Readiness-Driven Runtime for Pipeline-Parallel Training under Runtime Variability](https://mlsanity.com/paper/2605.18750/) (2026)

## Abstract (arXiv)

Current techniques and systems for distributed model training mostly assume that clusters are comprised of homogeneous servers with a constant resource availability. However, cluster heterogeneity is pervasive in computing infrastructure, and is a fundamental characteristic of low-cost transient resources (such as EC2 spot instances). In this paper, we develop a dynamic batching technique for distributed data-parallel training that adjusts the mini-batch sizes on each worker based on its resource availability and throughput. Our mini-batch controller seeks to equalize iteration times on all workers, and facilitates training on clusters comprised of servers with different amounts of CPU and GPU resources. This variable mini-batch technique uses proportional control and ideas from PID controllers to find stable mini-batch sizes. Our empirical evaluation shows that dynamic batching can reduce model training times by more than 4x on heterogeneous clusters.