mlsanityData to 14 Oct 2026

A Variational Perspective on Accelerated Methods in Optimization

arXiv 1603.04245 ↗First posted 14 Mar 2016math.OC cs.LG stat.MLcited by 110

Not on the map. Why?

Abstract (arXiv)

Accelerated gradient methods play a central role in optimization, achieving optimal rates in many settings. While many generalizations and extensions of Nesterov's original acceleration method have been proposed, it is not yet clear what is the natural scope of the acceleration concept. In this paper, we study accelerated methods from a continuous-time perspective. We show that there is a Lagrangian functional that we call the Bregman Lagrangian which generates a large class of accelerated methods in continuous time, including (but not limited to) accelerated gradient descent, its non-Euclidean extension, and accelerated higher-order gradient methods. We show that the continuous-time limit of all of these methods correspond to traveling the same curve in spacetime at different speeds. From this perspective, Nesterov's technique and many of its generalizations can be viewed as a systematic way to go from the continuous-time curves generated by the Bregman Lagrangian to a family of discrete-time accelerated algorithms.

Key facts. Cited by 110 papers in the map (data through 14 Oct 2026). First posted on arXiv 14 Mar 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 23; 2022: 21; 2023: 18; 2024: 17; 2025: 11; 2026: 20.

110 papers.

Sort:

By area:

  1. 2021Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networkscited by 50
  2. 2021Continuous vs. Discrete Optimization of Deep Neural Networkscited by 34
  3. 2024BELM: Bidirectional Explicit Linear Multi-step Sampler for Exact Inversion in Diffusion Modelscited by 29
  4. 2021The Confluence of Networks, Games and Learningcited by 26
  5. 2023Towards Predicting Equilibrium Distributions for Molecular Systems with Deep Learningcited by 26
  6. 2021No-Regret Dynamics in the Fenchel Game: A Unified Framework for Algorithmic Convex Optimizationcited by 20
  7. 2021A Continuized View on Nesterov Acceleration for Stochastic Gradient Descent and Randomized Gossipcited by 15
  8. 2021Discretization Drift in Two-Player Gamescited by 14
  9. 2021On Constraints in First-Order Optimization: A View from Non-Smooth Dynamical Systemscited by 13
  10. 2022Alternating Mirror Descent for Constrained Min-Max Gamescited by 13
  11. 2022Momentum Stiefel Optimizer, with Applications to Suitably-Orthogonal Attention, and Optimal Transportcited by 11
  12. 2024Keep the Momentum: Conservation Laws beyond Euclidean Gradient Flowscited by 11
38 more
  1. 2021How Does Momentum Benefit Deep Neural Networks Architecture Design? A Few Case Studiescited by 10
  2. 2023Accelerating optimization over the space of probability measurescited by 10
  3. 2026Momentum Further Constrains Sharpness at the Edge of Stochastic Stabilitycited by 10
  4. 2022On Quantum Speedups for Nonconvex Optimization via Quantum Tunneling Walkscited by 9
  5. 2022Towards Understanding GD with Hard and Conjugate Pseudo-labels for Test-Time Adaptationcited by 9
  6. 2023GAD-PVI: A General Accelerated Dynamic-Weight Particle-Based Variational Inference Frameworkcited by 9
  7. 2024Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networkscited by 9
  8. 2025Neural Thermodynamics: Entropic Forces in Deep and Universal Representation Learningcited by 9
  9. 2021Last-Iterate Convergence of Saddle-Point Optimizers via High-Resolution Differential Equationscited by 8
  10. 2022Gradient Norm Minimization of Nesterov Acceleration: o(1/k3)cited by 8
  11. 2022A Recursively Recurrent Neural Network (R2N2) Architecture for Learning Iterative Algorithmscited by 8
  12. 2021Convergence and Stability of the Stochastic Proximal Point Algorithm with Momentumcited by 7
  13. 2023Quantum Hamiltonian Descentcited by 7
  14. 2023Quantum Langevin Dynamics for Optimizationcited by 7
  15. 2024Modeling AdaGrad, RMSProp, and Adam with Integro-Differential Equationscited by 7
  16. 2025Hellinger-Kantorovich Gradient Flows: Global Exponential Decay of Entropy Functionalscited by 7
  17. 2021Revisiting the Role of Euler Numerical Integration on Acceleration and Stability in Convex Optimizationcited by 6
  18. 2021Optimization on manifolds: A symplectic approachcited by 6
  19. 2022Lyapunov function approach for approximation algorithm design and analysis: with applications in submodular maximizationcited by 6
  20. 2023Continuized Acceleration for Quasar Convex Functions in Non-Convex Optimizationcited by 6
  21. 2022Geometric Methods for Sampling, Optimisation, Inference and Adaptive Agentscited by 5
  22. 2022Perseus: A Simple and Optimal High-Order Method for Variational Inequalitiescited by 5
  23. 2022Riemannian accelerated gradient methods via extrapolationcited by 5
  24. 2023Accelerated First-Order Optimization under Nonlinear Constraintscited by 5
  25. 2023Sublinear Convergence Rates of Extragradient-Type Methods: A Survey on Classical and Recent Developmentscited by 5
  26. 2026Symmetry Breaking in Transformers for Efficient and Interpretable Trainingcited by 5
  27. 2021First-Order Methods for Convex Optimizationcited by 4
  28. 2021Potential Function-based Framework for Making the Gradients Small in Convex and Min-Max Optimizationcited by 4
  29. 2022Accelerated Riemannian Optimization: Handling Constraints with a Prox to Bound Geometric Penaltiescited by 4
  30. 2024Conformal Symplectic Optimization for Stable Reinforcement Learningcited by 4
  31. 2025Finite-Time Analysis of Discrete-Time Stochastic Interpolantscited by 4
  32. 2025Gradient Descent as Loss Landscape Navigation: a Normative Framework for Deriving Learning Rulescited by 4
  33. 2026Muon with Spectral Guidance: Efficient Optimization for Scientific Machine Learningcited by 4
  34. 2021Online Algorithms and Policies Using Adaptive and Machine Learning Approachescited by 3
  35. 2021A Contraction Theory Approach to Optimization Algorithms from Acceleration Flowscited by 3
  36. 2021A Deterministic Sampling Method via Maximum Mean Discrepancy Flow with Adaptive Kernelcited by 3
  37. 2021Breaking the Convergence Barrier: Optimization via Fixed-Time Convergent Flowscited by 3
  38. 2022Mean-field analysis for heavy ball methods: Dropout-stability, connectivity, and global convergencecited by 3

50 most cited of 110 All 110 (JSON)