# Coupling Adaptive Batch Sizes with Learning Rates

arXiv: https://arxiv.org/abs/1612.05086 · HTML: https://mlsanity.com/paper/1612.05086/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Dec 2016.

## Built on by (most cited first, 30 of 30)

- [Large Scale Private Learning via Low-rank Reparametrization](https://mlsanity.com/paper/2106.09352/) (2021)
- [Practical Efficiency of Muon for Pretraining](https://mlsanity.com/paper/2505.02222/) (2025)
- [Green Edge AI: A Contemporary Survey](https://mlsanity.com/paper/2312.00333/) (2023)
- [Surge Phenomenon in Optimal Learning Rate and Batch Size Scaling](https://mlsanity.com/paper/2405.14578/) (2024)
- [Adaptive shot allocation for fast convergence in variational quantum algorithms](https://mlsanity.com/paper/2108.10434/) (2021)
- [Increasing Both Batch Size and Learning Rate Accelerates Stochastic Gradient Descent](https://mlsanity.com/paper/2409.08770/) (2024)
- [AdAdaGrad: Adaptive Batch Size Schemes for Adaptive Gradient Methods](https://mlsanity.com/paper/2402.11215/) (2024)
- [Cockpit: A Practical Debugging Tool for the Training of Deep Neural Networks](https://mlsanity.com/paper/2102.06604/) (2021)
- [Stochastic Gradient Line Bayesian Optimization for Efficient Noise-Robust Optimization of Parameterized Quantum Circuits](https://mlsanity.com/paper/2111.07952/) (2021)
- [Normalization Layer Per-Example Gradients are Sufficient to Predict Gradient Noise Scale in Transformers](https://mlsanity.com/paper/2411.00999/) (2024)
- [Shot Optimization in Quantum Machine Learning Architectures to Accelerate Training](https://mlsanity.com/paper/2304.12950/) (2023)
- [Communication-Efficient Adaptive Batch Size Strategies for Distributed Local Gradient Methods](https://mlsanity.com/paper/2406.13936/) (2024)
- [Adaptive Batch Size Schedules for Distributed Training of Language Models with Data and Model Parallelism](https://mlsanity.com/paper/2412.21124/) (2024)
- [Unlocking optimal batch size schedules using continuous-time control and perturbation theory](https://mlsanity.com/paper/2312.01898/) (2023)
- [A statistical physics framework for optimal learning](https://mlsanity.com/paper/2507.07907/) (2025)
- [Adaptive Elastic Training for Sparse Deep Learning on Heterogeneous Multi-GPU Servers](https://mlsanity.com/paper/2110.07029/) (2021)
- [Using Stochastic Gradient Descent to Smooth Nonconvex Functions: Analysis of Implicit Graduated Optimization](https://mlsanity.com/paper/2311.08745/) (2023)
- [Increasing Batch Size Improves Convergence of Stochastic Gradient Descent with Momentum](https://mlsanity.com/paper/2501.08883/) (2025)
- [Multiple Importance Sampling for Stochastic Gradient Estimation](https://mlsanity.com/paper/2407.15525/) (2024)
- [Convergence of Sharpness-Aware Minimization Algorithms using Increasing Batch Size and Decaying Learning Rate](https://mlsanity.com/paper/2409.09984/) (2024)
- [SIM-Shapley: A Stable and Computationally Efficient Approach to Shapley Value Approximation](https://mlsanity.com/paper/2505.08198/) (2025)
- [Accelerating SGDM via Learning Rate and Batch Size Schedules: A Lyapunov-Based Analysis](https://mlsanity.com/paper/2508.03105/) (2025)
- [Towards joint scaling laws with optimal batch size schedules](https://mlsanity.com/paper/2607.27731/) (2026)
- [Bolstering Stochastic Gradient Descent with Model Building](https://mlsanity.com/paper/2111.07058/) (2021)
- [Adaptive Optimization with Examplewise Gradients](https://mlsanity.com/paper/2112.00174/) (2021)
- [Dynamic Batch Adaptation](https://mlsanity.com/paper/2208.00815/) (2022)
- [Evaluation of adaptive sampling methods in scenario generation for virtual safety impact assessment of pre-crash safety systems](https://mlsanity.com/paper/2503.00815/) (2025)
- [DIVEBATCH: Accelerating Model Training Through Gradient-Diversity Aware Batch Size Adaptation](https://mlsanity.com/paper/2509.16173/) (2025)
- [Adaptive directional gradients for parameterised quantum circuits](https://mlsanity.com/paper/2606.09734/) (2026)
- [Why Does Adaptive Batching Help LLM Pretraining? A Perspective from Unbounded Variance](https://mlsanity.com/paper/2610.02355/) (2026)

## Abstract (arXiv)

Mini-batch stochastic gradient descent and variants thereof have become standard for large-scale empirical risk minimization like the training of neural networks. These methods are usually used with a constant batch size chosen by simple empirical inspection. The batch size significantly influences the behavior of the stochastic optimization algorithm, though, since it determines the variance of the gradient estimates. This variance also changes over the optimization process; when using a constant batch size, stability and convergence is thus often enforced by means of a (manually tuned) decreasing learning rate schedule. We propose a practical method for dynamic batch size adaptation. It estimates the variance of the stochastic gradients and adapts the batch size to decrease the variance proportionally to the value of the objective function, removing the need for the aforementioned learning rate decrease. In contrast to recent related work, our algorithm couples the batch size to the learning rate, directly reflecting the known relationship between the two. On popular image classification benchmarks, our batch size adaptation yields faster optimization convergence, while simultaneously simplifying learning rate tuning. A TensorFlow implementation is available.