mlsanityData to 7 Oct 2026

Variational Stochastic Gradient Descent for Deep Neural Networks

arXiv 2404.06549 ↗First posted 9 Apr 2024cs.LG stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

Current state-of-the-art optimizers are adaptive gradient-based optimization methods such as Adam. Recently, there has been an increasing interest in formulating gradient-based optimizers in a probabilistic framework for better modeling the uncertainty of the gradients. Here, we propose to combine both approaches, resulting in the Variational Stochastic Gradient Descent (VSGD) optimizer. We model gradient updates as a probabilistic model and utilize stochastic variational inference (SVI) to derive an efficient and effective update rule. Further, we show how our VSGD method relates to other adaptive gradient-based optimizers like Adam. Lastly, we carry out experiments on two image classification datasets and four deep neural network architectures, where we show that VSGD outperforms Adam and SGD.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 27 of its 47 references that are in the map. First posted on arXiv 9 Apr 2024.

Builds on

27 of 47 references.

References · 27

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  2. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  3. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  4. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  5. 2019On the Convergence of Adam and Beyondcited by 834
  6. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  7. 2014Deep Learning in Neural Networks: An Overviewcited by 765
  8. 2013Generating Sequences With Recurrent Neural Networkscited by 549
  9. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 298
  10. 2018When Gaussian Process Meets Big Data: A Review of Scalable GPscited by 205
  11. 2022Patches Are All You Need?cited by 201
  12. 2019Practical Deep Learning with Bayesian Principlescited by 158
15 more
  1. 2017A Bayesian Perspective on Generalization and Stochastic Gradient Descentcited by 134
  2. 2018Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adamcited by 131
  3. 2018A Sufficient Condition for Convergences of Adam and RMSPropcited by 124
  4. 2017Improving Generalization Performance by Switching from Adam to SGDcited by 121
  5. 2020An Improved Analysis of Stochastic Gradient Descent with Momentumcited by 117
  6. 2018On the Power of Over-parametrization in Neural Networks with Quadratic Activationcited by 102
  7. 2020Is SGD a Bayesian sampler? Well, almostcited by 50
  8. 2023Kronecker-Factored Approximate Curvature for Modern Neural Network Architecturescited by 49
  9. 2017Revisiting Normalized Gradient Descent: Fast Evasion of Saddle Pointscited by 28
  10. 2023Simplifying Momentum-based Positive-definite Submanifold Optimization with Applications to Deep Learningcited by 10
  11. 2018Kalman Gradient Descent: Adaptive Variance Reduction in Stochastic Optimizationcited by 8
  12. 2021Structured second-order methods via natural gradient descentcited by 7
  13. 2019Bayesian interpretation of SGD as Ito processcited by 3
  14. 2019Convergence Analysis of a Momentum Algorithm with Adaptive Step Size for Non Convex Optimizationcited by 2
  15. 2022Bayesian Stochastic Gradient Descent for Stochastic Optimization with Streaming Input Datacited by 2

Built on by

Not cited yet.