# AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networks

arXiv: https://arxiv.org/abs/1712.02029 · HTML: https://mlsanity.com/paper/1712.02029/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Dec 2017.

## Built on by (most cited first, 29 of 29)

- [Towards Efficient and Scalable Sharpness-Aware Minimization](https://mlsanity.com/paper/2203.02714/) (2022)
- [A Distributed Data-Parallel PyTorch Implementation of the Distributed Shampoo Optimizer for Training Neural Networks At-Scale](https://mlsanity.com/paper/2309.06497/) (2023)
- [On the Utility of Gradient Compression in Distributed Training Systems](https://mlsanity.com/paper/2103.00543/) (2021)
- [Taming Resource Heterogeneity In Distributed ML Training With Dynamic Batching](https://mlsanity.com/paper/2305.12213/) (2023)
- [Concurrent Adversarial Learning for Large-Batch Training](https://mlsanity.com/paper/2106.00221/) (2021)
- [Modeling the Machine Learning Multiverse](https://mlsanity.com/paper/2206.05985/) (2022)
- [Large-Scale Deep Learning Optimizations: A Comprehensive Survey](https://mlsanity.com/paper/2111.00856/) (2021)
- [AdAdaGrad: Adaptive Batch Size Schemes for Adaptive Gradient Methods](https://mlsanity.com/paper/2402.11215/) (2024)
- [Training LLMs with Fault Tolerant HSDP on 100,000 GPUs](https://mlsanity.com/paper/2602.00277/) (2026)
- [Illustrious: an Open Advanced Illustration Model](https://mlsanity.com/paper/2409.19946/) (2024)
- [Fast Jacobian-Vector Product for Deep Networks](https://mlsanity.com/paper/2104.00219/) (2021)
- [Communication-Efficient Adaptive Batch Size Strategies for Distributed Local Gradient Methods](https://mlsanity.com/paper/2406.13936/) (2024)
- [Adaptive Batch Size Schedules for Distributed Training of Language Models with Data and Model Parallelism](https://mlsanity.com/paper/2412.21124/) (2024)
- [Towards Sustainable Deep Learning for Wireless Fingerprinting Localization](https://mlsanity.com/paper/2201.09071/) (2022)
- [Accelerating Large Batch Training via Gradient Signal to Noise Ratio (GSNR)](https://mlsanity.com/paper/2309.13681/) (2023)
- [A statistical physics framework for optimal learning](https://mlsanity.com/paper/2507.07907/) (2025)
- [Adaptive Batch Sizes Using Non-Euclidean Gradient Noise Scales for Stochastic Sign and Spectral Descent](https://mlsanity.com/paper/2602.03001/) (2026)
- [Resource-aware Deep Learning for Wireless Fingerprinting Localization](https://mlsanity.com/paper/2211.01759/) (2022)
- [Wirelessly Powered Federated Learning Networks: Joint Power Transfer, Data Sensing, Model Training, and Resource Allocation](https://mlsanity.com/paper/2308.04953/) (2023)
- [Optimal Batch Allocation for Wireless Federated Learning](https://mlsanity.com/paper/2404.02395/) (2024)
- [Computational Approaches to Arabic-English Code-Switching](https://mlsanity.com/paper/2410.13318/) (2024)
- [HAND: Hierarchical Attention Network for Multi-Scale Handwritten Document Recognition and Layout Analysis](https://mlsanity.com/paper/2412.18981/) (2024)
- [Simple and Effective Balance of Contrastive Losses](https://mlsanity.com/paper/2112.11743/) (2021)
- [Hyper-Learning for Gradient-Based Batch Size Adaptation](https://mlsanity.com/paper/2205.08231/) (2022)
- [Dynamic Batch Adaptation](https://mlsanity.com/paper/2208.00815/) (2022)
- [AccelAT: A Framework for Accelerating the Adversarial Training of Deep Neural Networks through Accuracy Gradient](https://mlsanity.com/paper/2210.06888/) (2022)
- [Evaluation of adaptive sampling methods in scenario generation for virtual safety impact assessment of pre-crash safety systems](https://mlsanity.com/paper/2503.00815/) (2025)
- [DIVEBATCH: Accelerating Model Training Through Gradient-Diversity Aware Batch Size Adaptation](https://mlsanity.com/paper/2509.16173/) (2025)
- [Democratizing AI: A Comparative Study in Deep Learning Efficiency and Future Trends in Computational Processing](https://mlsanity.com/paper/2603.20920/) (2026)

## Abstract (arXiv)

Training deep neural networks with Stochastic Gradient Descent, or its variants, requires careful choice of both learning rate and batch size. While smaller batch sizes generally converge in fewer training epochs, larger batch sizes offer more parallelism and hence better computational efficiency. We have developed a new training approach that, rather than statically choosing a single batch size for all epochs, adaptively increases the batch size during the training process. Our method delivers the convergence rate of small batch sizes while achieving performance similar to large batch sizes. We analyse our approach using the standard AlexNet, ResNet, and VGG networks operating on the popular CIFAR-10, CIFAR-100, and ImageNet datasets. Our results demonstrate that learning with adaptive batch sizes can improve performance by factors of up to 6.25 on 4 NVIDIA Tesla P100 GPUs while changing accuracy by less than 1% relative to training with fixed batch sizes.