mlsanityData to 7 Oct 2026

Accelerated Gradient Flow: Risk, Stability, and Implicit Regularization

arXiv 2201.08311 ↗First posted 20 Jan 2022stat.ML cs.LG math.OCcited by 0

Not on the map. Why?

Abstract (arXiv)

Acceleration and momentum are the de facto standard in modern applications of machine learning and optimization, yet the bulk of the work on implicit regularization focuses instead on unaccelerated methods. In this paper, we study the statistical risk of the iterates generated by Nesterov's accelerated gradient method and Polyak's heavy ball method, when applied to least squares regression, drawing several connections to explicit penalization. We carry out our analyses in continuous-time, allowing us to make sharper statements than in prior work, and revealing complex interactions between early stopping, stability, and the curvature of the loss function.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 32 of its 56 references that are in the map. First posted on arXiv 20 Jan 2022.

Builds on

32 of 56 references.

References · 32

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  3. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  4. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  5. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 547
  6. 2017Implicit Regularization in Matrix Factorizationcited by 318
  7. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 259
  8. 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 248
  9. 2021Deep learning: a statistical viewpointcited by 239
  10. 2015A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insightscited by 203
  11. 2020A Simple Convergence Proof of Adam and Adagradcited by 169
  12. 2015High-Dimensional Asymptotics of Prediction: Ridge Regression and Classificationcited by 146
20 more
  1. 2018A Sufficient Condition for Convergences of Adam and RMSPropcited by 124
  2. 2016A Variational Perspective on Accelerated Methods in Optimizationcited by 108
  3. 2014Analysis and Design of Optimization Algorithms via Integral Quadratic Constraintscited by 105
  4. 2018Understanding the Acceleration Phenomenon via High-Resolution Differential Equationscited by 68
  5. 2018A Continuous-Time View of Early Stopping for Least Squarescited by 67
  6. 2018Stochastic Gradient Descent on Separable Data: Exact Convergence with a Fixed Learning Ratecited by 63
  7. 2014Global convergence of the Heavy-ball method for convex optimizationcited by 60
  8. 2014Linear Coupling: An Ultimate Unification of Gradient and Mirror Descentcited by 58
  9. 2020The Implicit Regularization of Stochastic Gradient Flow for Least Squarescited by 58
  10. 2018Stability and Convergence Trade-off of Iterative Optimization Algorithmscited by 53
  11. 2016Unified Convergence Analysis of Stochastic Momentum Methods for Convex and Non-convex Optimizationcited by 34
  12. 2014iPiano: Inertial Proximal Algorithm for Non-Convex Optimizationcited by 30
  13. 2020When Does Preconditioning Help or Hurt Generalization?cited by 30
  14. 2016A Lyapunov Analysis of Momentum Methods in Optimizationcited by 28
  15. 2016Gradient Descent Converges to Minimizerscited by 25
  16. 2015A geometric alternative to Nesterov's accelerated gradient descentcited by 21
  17. 2019Implicit Regularization and Convergence for Weight Normalizationcited by 20
  18. 2020The Statistical Complexity of Early-Stopped Mirror Descentcited by 19
  19. 2019Implicit Regularization of Accelerated Methods in Hilbert Spacescited by 14
  20. 2019Geometry, Computation, and Optimality in Stochastic Optimizationcited by 9

Built on by

Not cited yet.