mlsanityData to 14 Oct 2026

SGEM: stochastic gradient with energy and momentum

arXiv 2208.02208 ↗First posted 3 Aug 2022cs.LG math.OCcited by 0

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose SGEM, Stochastic Gradient with Energy and Momentum, to solve a large class of general non-convex stochastic optimization problems, based on the AEGD method that originated in the work [AEGD: Adaptive Gradient Descent with Energy. arXiv: 2010.05109]. SGEM incorporates both energy and momentum at the same time so as to inherit their dual advantages. We show that SGEM features an unconditional energy stability property, and derive energy-dependent convergence rates in the general nonconvex stochastic setting, as well as a regret bound in the online convex setting. A lower threshold for the energy variable is also provided. Our experimental results show that SGEM converges faster than AEGD and generalizes better or at least as well as SGDM in training some deep neural networks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 25 of its 43 references that are in the map. First posted on arXiv 3 Aug 2022.

Builds on

25 of 43 references.

References · 25

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  5. 2016Densely Connected Convolutional Networkscited by 6,720
  6. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
  7. 2019On the Convergence of Adam and Beyondcited by 841
  8. 2019Introduction to Online Convex Optimizationcited by 803
  9. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 772
  10. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 539
  11. 2014SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectivescited by 402
  12. 2020AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradientscited by 283
13 more
  1. 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 250
  2. 2019Adaptive Gradient Methods with Dynamic Bound of Learning Ratecited by 183
  3. 2019On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimizationcited by 180
  4. 2018On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimizationcited by 174
  5. 2018On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizescited by 156
  6. 2016Katyusha: The First Direct Acceleration of Stochastic Gradient Methodscited by 135
  7. 2018A Sufficient Condition for Convergences of Adam and RMSPropcited by 126
  8. 2020An Improved Analysis of Stochastic Gradient Descent with Momentumcited by 119
  9. 2017Accelerated Gradient Descent Escapes Saddle Points Faster than Gradient Descentcited by 66
  10. 2017Non-convex Finite-Sum Optimization Via SCSG Methodscited by 61
  11. 2018A Unified Analysis of Stochastic Momentum Methods for Deep Learningcited by 55
  12. 2020AEGD: Adaptive Gradient Descent with Energycited by 6
  13. 2022Dynamic behavior for a gradient algorithm with energy and momentumcited by 2

Built on by

Not cited yet.