mlsanityData to 7 Oct 2026

AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networks

arXiv 1712.02029 ↗First posted 6 Dec 2017cs.LG cs.CV cs.DCcited by 29

Not on the map. Why?

Abstract (arXiv)

Training deep neural networks with Stochastic Gradient Descent, or its variants, requires careful choice of both learning rate and batch size. While smaller batch sizes generally converge in fewer training epochs, larger batch sizes offer more parallelism and hence better computational efficiency. We have developed a new training approach that, rather than statically choosing a single batch size for all epochs, adaptively increases the batch size during the training process. Our method delivers the convergence rate of small batch sizes while achieving performance similar to large batch sizes. We analyse our approach using the standard AlexNet, ResNet, and VGG networks operating on the popular CIFAR-10, CIFAR-100, and ImageNet datasets. Our results demonstrate that learning with adaptive batch sizes can improve performance by factors of up to 6.25 on 4 NVIDIA Tesla P100 GPUs while changing accuracy by less than 1% relative to training with fixed batch sizes.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). First posted on arXiv 6 Dec 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 7; 2023: 4; 2024: 7; 2025: 3; 2026: 3.

29 papers.

Sort:

  1. 2022Towards Efficient and Scalable Sharpness-Aware Minimizationcited by 129
  2. 2023A Distributed Data-Parallel PyTorch Implementation of the Distributed Shampoo Optimizer for Training Neural Networks At-Scalecited by 69
  3. 2021On the Utility of Gradient Compression in Distributed Training Systemscited by 26
  4. 2023Taming Resource Heterogeneity In Distributed ML Training With Dynamic Batchingcited by 12
  5. 2021Concurrent Adversarial Learning for Large-Batch Trainingcited by 11
  6. 2022Modeling the Machine Learning Multiversecited by 11
  7. 2021Large-Scale Deep Learning Optimizations: A Comprehensive Surveycited by 10
  8. 2024AdAdaGrad: Adaptive Batch Size Schemes for Adaptive Gradient Methodscited by 8
  9. 2026Training LLMs with Fault Tolerant HSDP on 100,000 GPUscited by 6
  10. 2024Illustrious: an Open Advanced Illustration Modelcited by 5
  11. 2021Fast Jacobian-Vector Product for Deep Networkscited by 4
  12. 2024Communication-Efficient Adaptive Batch Size Strategies for Distributed Local Gradient Methodscited by 4
17 more
  1. 2024Adaptive Batch Size Schedules for Distributed Training of Language Models with Data and Model Parallelismcited by 4
  2. 2022Towards Sustainable Deep Learning for Wireless Fingerprinting Localizationcited by 3
  3. 2023Accelerating Large Batch Training via Gradient Signal to Noise Ratio (GSNR)cited by 3
  4. 2025A statistical physics framework for optimal learningcited by 3
  5. 2022Resource-aware Deep Learning for Wireless Fingerprinting Localizationcited by 1
  6. 2023Wirelessly Powered Federated Learning Networks: Joint Power Transfer, Data Sensing, Model Training, and Resource Allocationcited by 1
  7. 2024Optimal Batch Allocation for Wireless Federated Learningcited by 1
  8. 2024Computational Approaches to Arabic-English Code-Switchingcited by 1
  9. 2024HAND: Hierarchical Attention Network for Multi-Scale Handwritten Document Recognition and Layout Analysiscited by 1
  10. 2021Simple and Effective Balance of Contrastive Lossescited by 0
  11. 2022Hyper-Learning for Gradient-Based Batch Size Adaptationcited by 0
  12. 2022Dynamic Batch Adaptationcited by 0
  13. 2022AccelAT: A Framework for Accelerating the Adversarial Training of Deep Neural Networks through Accuracy Gradientcited by 0
  14. 2025Evaluation of adaptive sampling methods in scenario generation for virtual safety impact assessment of pre-crash safety systemscited by 0
  15. 2025DIVEBATCH: Accelerating Model Training Through Gradient-Diversity Aware Batch Size Adaptationcited by 0
  16. 2026Adaptive Batch Sizes Using Non-Euclidean Gradient Noise Scales for Stochastic Sign and Spectral Descentcited by 0
  17. 2026Democratizing AI: A Comparative Study in Deep Learning Efficiency and Future Trends in Computational Processingcited by 0