mlsanityData to 7 Oct 2026

A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights

arXiv 1503.01243 ↗First posted 4 Mar 2015stat.ML math.CA math.OCcited by 203

Not on the map. Why?

Abstract (arXiv)

We derive a second-order ordinary differential equation (ODE) which is the limit of Nesterov's accelerated gradient method. This ODE exhibits approximate equivalence to Nesterov's scheme and thus can serve as a tool for analysis. We show that the continuous time ODE allows for a better understanding of Nesterov's scheme. As a byproduct, we obtain a family of schemes with similar convergence rates. The ODE interpretation also suggests restarting Nesterov's scheme leading to an algorithm, which can be rigorously proven to converge at a linear rate whenever the objective is strongly convex.

Key facts. Cited by 203 papers in the map (data through 7 Oct 2026). First posted on arXiv 4 Mar 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 38; 2022: 40; 2023: 33; 2024: 29; 2025: 26; 2026: 37.

203 papers.

Sort:

By area:

  1. 2022Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Modelscited by 84
  2. 2021Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networkscited by 47
  3. 2021Heavy Ball Neural Ordinary Differential Equationscited by 42
  4. 2021Continuous vs. Discrete Optimization of Deep Neural Networkscited by 34
  5. 2021SGD in the Large: Average-case Analysis, Asymptotics, and Stepsize Criticalitycited by 33
  6. 2021The Confluence of Networks, Games and Learningcited by 26
  7. 2022Sparse Neural Additive Model: Interpretable Deep Learning with Feature Selection via Group Sparsitycited by 25
  8. 2023An SDE for Modeling SAM: Theory and Insightscited by 22
  9. 2021Does Momentum Change the Implicit Regularization on Separable Data?cited by 21
  10. 2021No-Regret Dynamics in the Fenchel Game: A Unified Framework for Algorithmic Convex Optimizationcited by 20
  11. 2021Noether: The More Things Change, the More Stay the Samecited by 19
  12. 2021Edge Federated Learning Via Unit-Modulus Over-The-Air Computationcited by 16
38 more
  1. 2021A Continuized View on Nesterov Acceleration for Stochastic Gradient Descent and Randomized Gossipcited by 15
  2. 2021Discretization Drift in Two-Player Gamescited by 14
  3. 2021Acceleration via Fractal Learning Rate Schedulescited by 13
  4. 2021On Constraints in First-Order Optimization: A View from Non-Smooth Dynamical Systemscited by 13
  5. 2023Towards Understanding the Dynamics of Gaussian-Stein Variational Gradient Descentcited by 13
  6. 2022Continuous-Time Meta-Learning with Forward Mode Differentiationcited by 12
  7. 2022Toward Equation of Motion for Deep Neural Networks: Continuous-time Gradient Descent and Discretization Error Analysiscited by 12
  8. 2022Effects of Data Geometry in Early Deep Learningcited by 12
  9. 2024Keep the Momentum: Conservation Laws beyond Euclidean Gradient Flowscited by 11
  10. 2021How Does Momentum Benefit Deep Neural Networks Architecture Design? A Few Case Studiescited by 10
  11. 2021Convergence Rates of Two-Time-Scale Gradient Descent-Ascent Dynamics for Solving Nonconvex Min-Max Problemscited by 10
  12. 2022A duality connecting neural network and cosmological dynamicscited by 10
  13. 2022Momentum Stiefel Optimizer, with Applications to Suitably-Orthogonal Attention, and Optimal Transportcited by 10
  14. 2023Stochastic Approximation Beyond Gradient for Signal Processing and Machine Learningcited by 10
  15. 2021ANITA: An Optimal Loopless Accelerated Variance-Reduced Gradient Methodcited by 9
  16. 2021On the Hyperparameters in Stochastic Gradient Descent with Momentumcited by 9
  17. 2022On Quantum Speedups for Nonconvex Optimization via Quantum Tunneling Walkscited by 9
  18. 2023Accelerating optimization over the space of probability measurescited by 9
  19. 2024Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networkscited by 9
  20. 2021Last-Iterate Convergence of Saddle-Point Optimizers via High-Resolution Differential Equationscited by 8
  21. 2022Gradient Norm Minimization of Nesterov Acceleration: o(1/k3)cited by 8
  22. 2022A Recursively Recurrent Neural Network (R2N2) Architecture for Learning Iterative Algorithmscited by 8
  23. 2026Accelerating LLM Pre-Training through Flat-Direction Dynamics Enhancementcited by 8
  24. 2026Momentum Further Constrains Sharpness at the Edge of Stochastic Stabilitycited by 8
  25. 2021Provable Convergence of Nesterov's Accelerated Gradient Method for Over-Parameterized Neural Networkscited by 7
  26. 2021Convergence and Stability of the Stochastic Proximal Point Algorithm with Momentumcited by 7
  27. 2023Quantum Hamiltonian Descentcited by 7
  28. 2023Quantum Langevin Dynamics for Optimizationcited by 7
  29. 2021Revisiting the Role of Euler Numerical Integration on Acceleration and Stability in Convex Optimizationcited by 6
  30. 2022Lyapunov function approach for approximation algorithm design and analysis: with applications in submodular maximizationcited by 6
  31. 2022Proximal Subgradient Norm Minimization of ISTA and FISTAcited by 6
  32. 2024Nesterov acceleration in benignly non-convex landscapescited by 6
  33. 2024Modeling AdaGrad, RMSProp, and Adam with Integro-Differential Equationscited by 6
  34. 2021Analytical Study of Momentum-Based Acceleration Methods in Paradigmatic High-Dimensional Non-Convex Problemscited by 5
  35. 2022Geometric Methods for Sampling, Optimisation, Inference and Adaptive Agentscited by 5
  36. 2022Riemannian accelerated gradient methods via extrapolationcited by 5
  37. 2022From Gradient Flow on Population Loss to Learning with Stochastic Gradient Descentcited by 5
  38. 2023Accelerated First-Order Optimization under Nonlinear Constraintscited by 5

50 most cited of 203 All 203 (JSON)