mlsanityData to 7 Oct 2026

Non-Gaussianity of Stochastic Gradient Noise

arXiv 1910.09626 ↗First posted 21 Oct 2019cs.LG stat.MLcited by 22

Not on the map. Why?

Abstract (arXiv)

What enables Stochastic Gradient Descent (SGD) to achieve better generalization than Gradient Descent (GD) in Neural Network training? This question has attracted much attention. In this paper, we study the distribution of the Stochastic Gradient Noise (SGN) vectors during the training. We observe that for batch sizes 256 and above, the distribution is best described as Gaussian at-least in the early phases of training. This holds across data-sets, architectures, and other choices.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). First posted on arXiv 21 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 9; 2023: 2; 2024: 4; 2025: 1; 2026: 1.

22 papers.

Sort:

  1. 2022Anticorrelated Noise Injection for Improved Generalizationcited by 41
  2. 2021On Proximal Policy Optimization's Heavy-tailed Gradientscited by 30
  3. 2021Generalization Bounds using Lower Tail Exponents in Stochastic Optimizerscited by 27
  4. 2023An SDE for Modeling SAM: Theory and Insightscited by 22
  5. 2022On the Power-Law Hessian Spectrums in Deep Learningcited by 14
  6. 2022On the Overlooked Structure of Stochastic Gradientscited by 13
  7. 2022Two Facets of SDE Under an Information-Theoretic Lens: Generalization of SGD via Training Trajectories and via Terminal Statescited by 11
  8. 2022Taming Fat-Tailed ("Heavier-Tailed'' with Potentially Infinite Variance) Noise in Federated Learningcited by 10
  9. 2022Heavy-Tail Phenomenon in Decentralized SGDcited by 6
  10. 2021Exponential escape efficiency of SGD from sharp minima in non-stationary regimecited by 5
  11. 2022On generalization bounds for deep networks based on loss surface implicit regularizationcited by 5
  12. 2022On the Theoretical Properties of Noise Correlation in Stochastic Optimizationcited by 5
10 more
  1. 2023High Probability Analysis for Non-Convex Stochastic Optimization with Clippingcited by 5
  2. 2024Lossless and Near-Lossless Compression for Foundation Modelscited by 5
  3. 2021Revisiting the Characteristics of Stochastic Gradient Noise and Dynamicscited by 4
  4. 2024Clip Body and Tail Separately: High Probability Guarantees for DPSGD with Heavy Tailscited by 4
  5. 2024Can LLMs predict the convergence of Stochastic Gradient Descent?cited by 2
  6. 2025Variational Learning Finds Flatter Solutions at the Edge of Stabilitycited by 2
  7. 2021Understanding Short-Range Memory Effects in Deep Neural Networkscited by 1
  8. 2022Biases in In Silico Evaluation of Molecular Optimization Methods and Bias-Reduced Evaluation Methodologycited by 1
  9. 2024Distributed Stochastic Gradient Descent with Staleness: A Stochastic Delay Differential Equation Based Frameworkcited by 1
  10. 2026Clipping Makes Distributed and Federated Asynchronous SGD Robust to Stragglerscited by 0