# First and Second Order Approximations to Stochastic Gradient Descent Methods with Momentum Terms

arXiv: https://arxiv.org/abs/2504.13992 · HTML: https://mlsanity.com/paper/2504.13992/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 3 of its 8 references that are in the map.
- First posted on arXiv 18 Apr 2025.

## Builds on (most cited first, 3 of 3)

- [Handbook of Convergence Theorems for (Stochastic) Gradient Methods](https://mlsanity.com/paper/2301.11235/) (2023)
- [An Improved Analysis of Stochastic Gradient Descent with Momentum](https://mlsanity.com/paper/2007.07989/) (2020)
- [The ODE Method for Stochastic Approximation and Reinforcement Learning with Markovian Noise](https://mlsanity.com/paper/2401.07844/) (2024)

## Abstract (arXiv)

Stochastic Gradient Descent (SGD) methods see many uses in optimization problems. Modifications to the algorithm, such as momentum-based SGD methods have been known to produce better results in certain cases. Much of this, however, is due to empirical information rather than rigorous proof. While the dynamics of gradient descent methods can be studied through continuous approximations, existing works only cover scenarios with constant learning rates or SGD without momentum terms. We present approximation results under weak assumptions for SGD that allow learning rates and momentum parameters to vary with respect to time.