mlsanityData to 7 Oct 2026

Gradient Descent-Type Methods: Background and Simple Unified Convergence Analysis

arXiv 2212.09413 ↗First posted 19 Dec 2022math.OC stat.MLcited by 1

Not on the map. Why?

Abstract (arXiv)

In this book chapter, we briefly describe the main components that constitute the gradient descent method and its accelerated and stochastic variants. We aim at explaining these components from a mathematical point of view, including theoretical and practical aspects, but at an elementary level. We will focus on basic variants of the gradient descent method and then extend our view to recent variants, especially variance-reduced stochastic gradient schemes (SGD). Our approach relies on revealing the structures presented inside the problem and the assumptions imposed on the objective function. Our convergence analysis unifies several known results and relies on a general, but elementary recursive expression. We have illustrated this analysis on several common schemes.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 24 of its 68 references that are in the map. First posted on arXiv 19 Dec 2022.

Builds on

24 of 68 references.

References · 24

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  3. 2021Convex optimizationcited by 2,479
  4. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  5. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 716
  6. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 526
  7. 2014SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectivescited by 401
  8. 2019Momentum-Based Variance Reduction in Non-Convex SGDcited by 302
  9. 2015Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decompositioncited by 241
  10. 2017SARAH: A Novel Method for Machine Learning Problems Using Stochastic Recursive Gradientcited by 218
  11. 2015A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insightscited by 203
  12. 2013Minimizing Finite Sums with the Stochastic Average Gradientcited by 164
12 more
  1. 2019Don't Jump Through Hoops and Remove Those Loops: SVRG and Katyusha are Better Without the Outer Loopcited by 68
  2. 2021Fast Extra Gradient Methods for Smooth Structured Nonconvex-Nonconcave Minimax Problemscited by 61
  3. 2016Error bounds, quadratic growth, and linear convergence of proximal methodscited by 59
  4. 2019ProxSARAH: An Efficient Algorithmic Framework for Stochastic Composite Nonconvex Optimizationcited by 37
  5. 2020SMG: A Shuffling Gradient-Based Method with Momentumcited by 22
  6. 2022From Halpern's Fixed-Point Iterations to Nesterov's Accelerated Interpretations for Root-Finding Problemscited by 18
  7. 2019On the Convergence of SARAH and Beyondcited by 10
  8. 2019Accelerating Variance-Reduced Stochastic Gradient Methodscited by 10
  9. 2013Composite Self-Concordant Minimizationcited by 8
  10. 2020Hybrid Variance-Reduced SGD Algorithms For Nonconvex-Concave Minimax Problemscited by 4
  11. 2016Restarting accelerated gradient methods with a rough strong convexity estimatecited by 3
  12. 2018Characterization of Convex Objective Functions and Optimal Expected Convergence Rates for SGDcited by 1

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024Learning dynamical systems from data: Gradient-based dictionary optimizationcited by 3