# Revisiting the Characteristics of Stochastic Gradient Noise and Dynamics

arXiv: https://arxiv.org/abs/2109.09833 · HTML: https://mlsanity.com/paper/2109.09833/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 60 references that are in the map.
- First posted on arXiv 20 Sep 2021.

## Builds on (most cited first, 19 of 19)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [A Survey of Deep Learning Techniques for Autonomous Driving](https://mlsanity.com/paper/1910.07738/) (2019)
- [Stochastic Gradient Descent as Approximate Bayesian Inference](https://mlsanity.com/paper/1704.04289/) (2017)
- [A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks](https://mlsanity.com/paper/1901.06053/) (2019)
- [Stochastic modified equations and adaptive stochastic gradient algorithms](https://mlsanity.com/paper/1511.06251/) (2015)
- [SMARTS: Scalable Multi-Agent Reinforcement Learning Training School for Autonomous Driving](https://mlsanity.com/paper/2010.09776/) (2020)
- [Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model](https://mlsanity.com/paper/1907.04164/) (2019)
- [Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games](https://mlsanity.com/paper/1703.10069/) (2017)
- [Fractional Underdamped Langevin Dynamics: Retargeting SGD with Momentum under Heavy-Tailed Gradient Noise](https://mlsanity.com/paper/2002.05685/) (2020)
- [Non-Gaussianity of Stochastic Gradient Noise](https://mlsanity.com/paper/1910.09626/) (2019)
- [Dynamic of Stochastic Gradient Descent with State-Dependent Noise](https://mlsanity.com/paper/2006.13719/) (2020)
- [Can Deep Learning Predict Risky Retail Investors? A Case Study in Financial Risk Behavior Forecasting](https://mlsanity.com/paper/1812.06175/) (2018)
- [Thermostat-assisted continuously-tempered Hamiltonian Monte Carlo for Bayesian learning](https://mlsanity.com/paper/1711.11511/) (2017)
- [On the Noisy Gradient Descent that Generalizes as SGD](https://mlsanity.com/paper/1906.07405/) (2019)
- [Benchmarking Deep Sequential Models on Volatility Predictions for Financial Time Series](https://mlsanity.com/paper/1811.03711/) (2018)
- [Replica-exchange Nos'e-Hoover dynamics for Bayesian learning on large datasets](https://mlsanity.com/paper/1905.12569/) (2019)

## Built on by (most cited first, 4 of 4)

- [On uniform-in-time diffusion approximation for stochastic gradient descent](https://mlsanity.com/paper/2207.04922/) (2022)
- [Robust Adaptation of Foundation Models with Black-Box Visual Prompting](https://mlsanity.com/paper/2407.17491/) (2024)
- [Weight fluctuations in (deep) linear neural networks and a derivation of the inverse-variance flatness relation](https://mlsanity.com/paper/2311.14120/) (2023)
- [OptEx: Expediting First-Order Optimization with Approximately Parallelized Iterations](https://mlsanity.com/paper/2402.11427/) (2024)

## Abstract (arXiv)

In this paper, we characterize the noise of stochastic gradients and analyze the noise-induced dynamics during training deep neural networks by gradient-based optimizers. Specifically, we firstly show that the stochastic gradient noise possesses finite variance, and therefore the classical Central Limit Theorem (CLT) applies; this indicates that the gradient noise is asymptotically Gaussian. Such an asymptotic result validates the wide-accepted assumption of Gaussian noise. We clarify that the recently observed phenomenon of heavy tails within gradient noise may not be intrinsic properties, but the consequence of insufficient mini-batch size; the gradient noise, which is a sum of limited i.i.d. random variables, has not reached the asymptotic regime of CLT, thus deviates from Gaussian. We quantitatively measure the goodness of Gaussian approximation of the noise, which supports our conclusion. Secondly, we analyze the noise-induced dynamics of stochastic gradient descent using the Langevin equation, granting for momentum hyperparameter in the optimizer with a physical interpretation. We then proceed to demonstrate the existence of the steady-state distribution of stochastic gradient descent and approximate the distribution at a small learning rate.