# AEGD: Adaptive Gradient Descent with Energy

arXiv: https://arxiv.org/abs/2010.05109 · HTML: https://mlsanity.com/paper/2010.05109/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Oct 2020.

## Built on by (most cited first, 6 of 6)

- [Muon with Spectral Guidance: Efficient Optimization for Scientific Machine Learning](https://mlsanity.com/paper/2602.16167/) (2026)
- [An Adaptive Gradient Method with Energy and Momentum](https://mlsanity.com/paper/2203.12191/) (2022)
- [A New Adaptive Gradient Method with Gradient Decomposition](https://mlsanity.com/paper/2107.08377/) (2021)
- [SGEM: stochastic gradient with energy and momentum](https://mlsanity.com/paper/2208.02208/) (2022)
- [An Element-wise RSAV Algorithm for Unconstrained Optimization Problems](https://mlsanity.com/paper/2309.04013/) (2023)
- [Accelerating Particle-based Energetic Variational Inference](https://mlsanity.com/paper/2504.03158/) (2025)

## Abstract (arXiv)

We propose AEGD, a new algorithm for first-order gradient-based optimization of non-convex objective functions, based on a dynamically updated energy variable. The method is shown to be unconditionally energy stable, irrespective of the step size. We prove energy-dependent convergence rates of AEGD for both non-convex and convex objectives, which for a suitably small step size recovers desired convergence rates for the batch gradient descent. We also provide an energy-dependent bound on the stationary convergence of AEGD in the stochastic non-convex setting. The method is straightforward to implement and requires little tuning of hyper-parameters. Experimental results demonstrate that AEGD works well for a large variety of optimization problems: it is robust with respect to initial data, capable of making rapid initial progress. The stochastic AEGD shows comparable and often better generalization performance than SGD with momentum for deep neural networks.