# The Physical Systems Behind Optimization Algorithms

arXiv: https://arxiv.org/abs/1612.02803 · HTML: https://mlsanity.com/paper/1612.02803/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Dec 2016.

## Built on by (most cited first, 2 of 2)

- [Computing the Variance of Shuffling Stochastic Gradient Algorithms via Power Spectral Density Analysis](https://mlsanity.com/paper/2206.00632/) (2022)
- [Understanding Accelerated Gradient Methods: Lyapunov Analyses and Hamiltonian Assisted Interpretations](https://mlsanity.com/paper/2304.10063/) (2023)

## Abstract (arXiv)

We use differential equations based approaches to provide some it physics insights into analyzing the dynamics of popular optimization algorithms in machine learning. In particular, we study gradient descent, proximal gradient descent, coordinate gradient descent, proximal coordinate gradient, and Newton's methods as well as their Nesterov's accelerated variants in a unified framework motivated by a natural connection of optimization algorithms to physical systems. Our analysis is applicable to more general algorithms and optimization problems it beyond convexity and strong convexity, e.g. Polyak-L ojasiewicz and error bound conditions (possibly nonconvex).