# Dynamic Batch Adaptation

arXiv: https://arxiv.org/abs/2208.00815 · HTML: https://mlsanity.com/paper/2208.00815/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 13 references that are in the map.
- First posted on arXiv 1 Aug 2022.

## Builds on (most cited first, 10 of 10)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Opacus: User-Friendly Differential Privacy Library in PyTorch](https://mlsanity.com/paper/2109.12298/) (2021)
- [Don't Decay the Learning Rate, Increase the Batch Size](https://mlsanity.com/paper/1711.00489/) (2017)
- [Coupling Adaptive Batch Sizes with Learning Rates](https://mlsanity.com/paper/1612.05086/) (2016)
- [AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networks](https://mlsanity.com/paper/1712.02029/) (2017)
- [Balancing Rates and Variance via Adaptive Batch-Size for Stochastic Optimization Problems](https://mlsanity.com/paper/2007.01219/) (2020)
- [Adaptive Learning of the Optimal Batch Size of SGD](https://mlsanity.com/paper/2005.01097/) (2020)
- [Estimation of discrete choice models with hybrid stochastic adaptive batch size algorithms](https://mlsanity.com/paper/2012.12155/) (2020)

## Abstract (arXiv)

Current deep learning adaptive optimizer methods adjust the step magnitude of parameter updates by altering the effective learning rate used by each parameter. Motivated by the known inverse relation between batch size and learning rate on update step magnitudes, we introduce a novel training procedure that dynamically decides the dimension and the composition of the current update step. Our procedure, Dynamic Batch Adaptation (DBA) analyzes the gradients of every sample and selects the subset that best improves certain metrics such as gradient variance for each layer of the network. We present results showing DBA significantly improves the speed of model convergence. Additionally, we find that DBA produces an increased improvement over standard optimizers when used in data scarce conditions where, in addition to convergence speed, it also significantly improves model generalization, managing to train a network with a single fully connected hidden layer using only 1% of the MNIST dataset to reach 97.79% test accuracy. In an even more extreme scenario, it manages to reach 97.44% test accuracy using only 10 samples per class. These results represent a relative error rate reduction of 81.78% and 88.07% respectively, compared to the standard optimizers, Stochastic Gradient Descent (SGD) and Adam.