mlsanityData to 7 Oct 2026

A Variational Perspective on High-Resolution ODEs

arXiv 2311.02002 ↗First posted 3 Nov 2023math.OC cs.LG cs.NAcited by 2

Not on the map. Why?

Abstract (arXiv)

We consider unconstrained minimization of smooth convex functions. We propose a novel variational perspective using forced Euler-Lagrange equation that allows for studying high-resolution ODEs. Through this, we obtain a faster convergence rate for gradient norm minimization using Nesterov's accelerated gradient method. Additionally, we show that Nesterov's method can be interpreted as a rate-matching discretization of an appropriately chosen high-resolution ODE. Finally, using the results from the new variational perspective, we propose a stochastic method for noisy gradients. Several numerical experiments compare and illustrate our stochastic algorithm with state of the art methods.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 22 of its 34 references that are in the map. First posted on arXiv 3 Nov 2023.

Builds on

22 of 34 references.

References · 22

  1. 2014SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectivescited by 401
  2. 2015A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insightscited by 203
  3. 2013Minimizing Finite Sums with the Stochastic Average Gradientcited by 164
  4. 2016A Variational Perspective on Accelerated Methods in Optimizationcited by 108
  5. 2014Analysis and Design of Optimization Algorithms via Integral Quadratic Constraintscited by 105
  6. 2018Understanding the Acceleration Phenomenon via High-Resolution Differential Equationscited by 68
  7. 2014Linear Coupling: An Ultimate Unification of Gradient and Mirror Descentcited by 58
  8. 2022Efficiently Computing Local Lipschitz Constants of Neural Networks via Bound Propagationcited by 45
  9. 2019A Dynamical Systems Perspective on Nesterov Accelerationcited by 33
  10. 2019Acceleration via Symplectic Discretization of High-Resolution Differential Equationscited by 27
  11. 2017Dissipativity Theory for Nesterov's Accelerated Methodcited by 22
  12. 2019First-order optimization algorithms via inertial systems with Hessian driven dampingcited by 22
10 more
  1. 2017Analysis of Optimization Algorithms via Integral Quadratic Constraints: Nonstrongly Convex Problemscited by 19
  2. 2019Nesterov's method with decreasing learning rate leads to accelerated stochastic gradient descentcited by 13
  3. 2021On Constraints in First-Order Optimization: A View from Non-Smooth Dynamical Systemscited by 13
  4. 2020The connections between Lyapunov functions for some optimization algorithms and differential equationscited by 11
  5. 2022Gradient Norm Minimization of Nesterov Acceleration: o(1/k3)cited by 8
  6. 2020Understanding Nesterov's Acceleration via Proximal Point Methodcited by 7
  7. 2019Accelerated First-Order Methods: Differential Equations and Lyapunov Functionscited by 5
  8. 2023Accelerated First-Order Optimization under Nonlinear Constraintscited by 5
  9. 2021Convergence of iterates for first-order optimization algorithms with inertia and Hessian driven dampingcited by 3
  10. 2020High-Resolution Modeling of the Fastest First-Order Optimization Method for Strongly Convex Functionscited by 2

Built on by

Cited by year

Citing papers per year: 2024: 2.

2 papers.

  1. 2024Architecture-Aware Learning Curve Extrapolation via Graph Ordinary Differential Equationcited by 5
  2. 2024Generalized Continuous-Time Models for Nesterov's Accelerated Gradient Methodscited by 0