# A Study of Gradient Variance in Deep Learning

arXiv: https://arxiv.org/abs/2007.04532 · HTML: https://mlsanity.com/paper/2007.04532/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jul 2020.

## Built on by (most cited first, 28 of 28)

- [Fishr: Invariant Gradient Variances for Out-of-Distribution Generalization](https://mlsanity.com/paper/2109.02934/) (2021)
- [An Experimental Study of Byzantine-Robust Aggregation Schemes in Federated Learning](https://mlsanity.com/paper/2302.07173/) (2023)
- [On the Generalization of Models Trained with SGD: Information-Theoretic Bounds and Implications](https://mlsanity.com/paper/2110.03128/) (2021)
- [Metadata Archaeology: Unearthing Data Subsets by Leveraging Training Dynamics](https://mlsanity.com/paper/2209.10015/) (2022)
- [A Tale Of Two Long Tails](https://mlsanity.com/paper/2107.13098/) (2021)
- [On the Limitations of Compute Thresholds as a Governance Strategy](https://mlsanity.com/paper/2407.05694/) (2024)
- [Critical Learning Periods: Leveraging Early Training Dynamics for Efficient Data Pruning](https://mlsanity.com/paper/2405.19462/) (2024)
- [Cockpit: A Practical Debugging Tool for the Training of Deep Neural Networks](https://mlsanity.com/paper/2102.06604/) (2021)
- [Low-Variance Forward Gradients using Direct Feedback Alignment and Momentum](https://mlsanity.com/paper/2212.07282/) (2022)
- [Normalization Layer Per-Example Gradients are Sufficient to Predict Gradient Noise Scale in Transformers](https://mlsanity.com/paper/2411.00999/) (2024)
- [Rethinking Adam: A Twofold Exponential Moving Average Approach](https://mlsanity.com/paper/2106.11514/) (2021)
- [Stable Language Model Pre-training by Reducing Embedding Variability](https://mlsanity.com/paper/2409.07787/) (2024)
- [On the Interpretability of Regularisation for Neural Networks Through Model Gradient Similarity](https://mlsanity.com/paper/2205.12642/) (2022)
- [Data-Centric Diet: Effective Multi-center Dataset Pruning for Medical Image Segmentation](https://mlsanity.com/paper/2308.01189/) (2023)
- [Per-example gradients: a new frontier for understanding and improving optimizers](https://mlsanity.com/paper/2510.00236/) (2025)
- [Layer-Aware Influence for Online Data Valuation Estimation](https://mlsanity.com/paper/2510.16007/) (2025)
- [Online Importance Sampling for Stochastic Gradient Optimization](https://mlsanity.com/paper/2311.14468/) (2023)
- [Multiple Importance Sampling for Stochastic Gradient Estimation](https://mlsanity.com/paper/2407.15525/) (2024)
- [Data value estimation on private gradients](https://mlsanity.com/paper/2412.17008/) (2024)
- [MSTGD:A Memory Stochastic sTratified Gradient Descent Method with an Exponential Convergence Rate](https://mlsanity.com/paper/2202.10923/) (2022)
- [Grad Queue : A probabilistic framework to reinforce sparse gradients](https://mlsanity.com/paper/2404.16917/) (2024)
- [FedDuA: Doubly Adaptive Federated Learning](https://mlsanity.com/paper/2505.11126/) (2025)
- [Insights from Gradient Dynamics: Gradient Autoscaled Normalization](https://mlsanity.com/paper/2509.03677/) (2025)
- [Rethinking the Harmonic Loss via Non-Euclidean Distance Layers](https://mlsanity.com/paper/2603.10225/) (2026)
- [LiteMatch: Lightweight Zero-Shot Stereo Matching via Cost Volume Stabilization](https://mlsanity.com/paper/2606.31636/) (2026)
- [Progressive2: A Teacher-Student Progressive Co-Evolving Knowledge Distillation Method for Substantial Model Compression](https://mlsanity.com/paper/2608.00129/) (2026)
- [Let Training Guide Selection: Online Synthetic Data Filtering via Real-Anchored Utility](https://mlsanity.com/paper/2609.29988/) (2026)
- [Efficient Neural Field Learning via Adaptive Coverage and Focused Sampling](https://mlsanity.com/paper/2610.02410/) (2026)

## Abstract (arXiv)

The impact of gradient noise on training deep models is widely acknowledged but not well understood. In this context, we study the distribution of gradients during training. We introduce a method, Gradient Clustering, to minimize the variance of average mini-batch gradient with stratified sampling. We prove that the variance of average mini-batch gradient is minimized if the elements are sampled from a weighted clustering in the gradient space. We measure the gradient variance on common deep learning benchmarks and observe that, contrary to common assumptions, gradient variance increases during training, and smaller learning rates coincide with higher variance. In addition, we introduce normalized gradient variance as a statistic that better correlates with the speed of convergence compared to gradient variance.