# SGD momentum optimizer with step estimation by online parabola model

arXiv: https://arxiv.org/abs/1907.07063 · HTML: https://mlsanity.com/paper/1907.07063/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jul 2019.

## Built on by (most cited first, 3 of 3)

- [Exploring the Relationship between Brain Hemisphere States and Frequency Bands through Classical Machine Learning and Deep Learning Optimization Techniques with Neurofeedback](https://mlsanity.com/paper/2509.14078/) (2025)
- [Comparing BFGS and OGR for Second-Order Optimization](https://mlsanity.com/paper/2512.06969/) (2025)
- [Effect of Activation Function and Model Optimizer on the Performance of Human Activity Recognition System Using Various Deep Learning Models](https://mlsanity.com/paper/2512.20104/) (2025)

## Abstract (arXiv)

In stochastic gradient descent, especially for neural network training, there are currently dominating first order methods: not modeling local distance to minimum. This information required for optimal step size is provided by second order methods, however, they have many difficulties, starting with full Hessian having square of dimension number of coefficients. This article proposes a minimal step from successful first order momentum method toward second order: online parabola modelling in just a single direction: normalized hatv from momentum method. It is done by estimating linear trend of gradients vecg=nabla F(vecθ) in hatv direction: such that g(vecθ_bot+θhatv)≈ λ (θ -p) for θ = vecθ· hatv, g= vecg· hatv, vecθ_bot=vecθ-θhatv. Using linear regression, λ, p are MSE estimated by just updating four averages (of g, θ, gθ, θ2) in the considered direction. Exponential moving averages allow here for inexpensive online estimation, weakening contribution of the old gradients. Controlling sign of curvature λ, we can repel from saddles in contrast to attraction in standard Newton method. In the remaining directions: not considered in second order model, we can simultaneously perform e.g. gradient descent. There is also discussed its learning rate approximation as μ=σθ / σg, allowing e.g. for adaptive SGD - with learning rate separately optimized (2nd order) for each parameter.