# The Impact of the Mini-batch Size on the Variance of Gradients in Stochastic Gradient Descent

arXiv: https://arxiv.org/abs/2004.13146 · HTML: https://mlsanity.com/paper/2004.13146/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Apr 2020.

## Built on by (most cited first, 16 of 16)

- [Green, Quantized Federated Learning over Wireless Networks: An Energy-Efficient Design](https://mlsanity.com/paper/2207.09387/) (2022)
- [Disparity Between Batches as a Signal for Early Stopping](https://mlsanity.com/paper/2107.06665/) (2021)
- [An Automatic Learning Rate Schedule Algorithm for Achieving Faster Convergence and Steeper Descent](https://mlsanity.com/paper/2310.11291/) (2023)
- [Divergence Results and Convergence of a Variance Reduced Version of ADAM](https://mlsanity.com/paper/2210.05607/) (2022)
- [EPINN-NSE: Enhanced Physics-Informed Neural Networks for Solving Navier-Stokes Equations](https://mlsanity.com/paper/2304.03689/) (2023)
- [Efficient NeRF Optimization -- Not All Samples Remain Equally Hard](https://mlsanity.com/paper/2408.03193/) (2024)
- [Adaptive posterior distributions for uncertainty analysis of covariance matrices in Bayesian inversion problems for multioutput signals](https://mlsanity.com/paper/2501.01148/) (2025)
- [SIM-Shapley: A Stable and Computationally Efficient Approach to Shapley Value Approximation](https://mlsanity.com/paper/2505.08198/) (2025)
- [MoRel: Long-Range Flicker-Free 4D Motion Modeling via Anchor Relay-based Bidirectional Blending with Hierarchical Densification](https://mlsanity.com/paper/2512.09270/) (2025)
- [Olivia: Harmonizing Time Series Foundation Models with Power Spectral Density](https://mlsanity.com/paper/2605.17340/) (2026)
- [Distributed Multi-Agent Reinforcement Learning with One-hop Neighbors and Compute Straggler Mitigation](https://mlsanity.com/paper/2202.09019/) (2022)
- [Privacy-Preserving Wavelet Neural Network with Fully Homomorphic Encryption](https://mlsanity.com/paper/2205.13265/) (2022)
- [Read the Signs: Towards Invariance to Gradient Descent's Hyperparameter Initialization](https://mlsanity.com/paper/2301.10133/) (2023)
- [Manifold Regularization for Memory-Efficient Training of Deep Neural Networks](https://mlsanity.com/paper/2305.17119/) (2023)
- [Robust Clustering on High-Dimensional Data with Stochastic Quantization](https://mlsanity.com/paper/2409.02066/) (2024)
- [A Trainable Optimizer](https://mlsanity.com/paper/2508.01764/) (2025)

## Abstract (arXiv)

The mini-batch stochastic gradient descent (SGD) algorithm is widely used in training machine learning models, in particular deep learning models. We study SGD dynamics under linear regression and two-layer linear networks, with an easy extension to deeper linear networks, by focusing on the variance of the gradients, which is the first study of this nature. In the linear regression case, we show that in each iteration the norm of the gradient is a decreasing function of the mini-batch size b and thus the variance of the stochastic gradient estimator is a decreasing function of b. For deep neural networks with L2 loss we show that the variance of the gradient is a polynomial in 1/b. The results back the important intuition that smaller batch sizes yield lower loss function values which is a common believe among the researchers. The proof techniques exhibit a relationship between stochastic gradient estimators and initial weights, which is useful for further research on the dynamics of SGD. We empirically provide further insights to our results on various datasets and commonly used deep network structures.