mlsanityData to 7 Oct 2026

Non-Asymptotic Analysis of Online Multiplicative Stochastic Gradient Descent

arXiv 2112.07110 ↗First posted 14 Dec 2021stat.ML cs.LG math.STcited by 0

Not on the map. Why?

Abstract (arXiv)

Past research has indicated that the covariance of the Stochastic Gradient Descent (SGD) error done via minibatching plays a critical role in determining its regularization and escape from low potential points. Motivated by some new research in this area, we prove universality results by showing that noise classes that have the same mean and covariance structure of SGD via minibatching have similar properties. We mainly consider the Multiplicative Stochastic Gradient Descent (M-SGD) algorithm as introduced in previous work, which has a much more general noise class than the SGD algorithm done via minibatching. We establish non asymptotic bounds for the M-SGD algorithm in the Wasserstein distance. We also show that the M-SGD error is approximately a scaled Gaussian distribution with mean 0 at any fixed point of the M-SGD algorithm.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 14 of its 37 references that are in the map. First posted on arXiv 14 Dec 2021.

Builds on

14 of 37 references.

References · 14

  1. 2021Convex optimizationcited by 2,479
  2. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  3. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  4. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 716
  5. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 278
  6. 2019Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networkscited by 221
  7. 2014Theoretical guarantees for approximate sampling from smooth and log-concave densitiescited by 206
  8. 2017Stochastic gradient descent performs variational inference, converges to limit cycles for deep networkscited by 108
  9. 2017Bridging the Gap between Constant Step Size Stochastic Gradient Descent and Markov Chainscited by 84
  10. 2020On the Almost Sure Convergence of Stochastic Gradient Descent in Non-Convex Problemscited by 79
  11. 2018Escaping Saddles with Stochastic Gradientscited by 66
  12. 2020An Analysis of Constant Step Size SGD in the Non-convex Regime: Asymptotic Normality and Biascited by 45
2 more
  1. 2019SGD Converges to Global Minimum in Deep Learning via Star-convex Pathcited by 39
  2. 2018On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descentcited by 25

Built on by

Not cited yet.