# CoolMomentum: A Method for Stochastic Optimization by Langevin Dynamics with Simulated Annealing

arXiv: https://arxiv.org/abs/2005.14605 · HTML: https://mlsanity.com/paper/2005.14605/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 May 2020.

## Built on by (most cited first, 4 of 4)

- [Equality Constrained Diffusion for Direct Trajectory Optimization](https://mlsanity.com/paper/2410.01939/) (2024)
- [ELRA: Exponential learning rate adaption gradient descent optimization method](https://mlsanity.com/paper/2309.06274/) (2023)
- [The Expected Loss of Preconditioned Langevin Dynamics Reveals the Hessian Rank](https://mlsanity.com/paper/2402.13810/) (2024)
- [Application of Langevin Dynamics to Advance the Quantum Natural Gradient Optimization Algorithm](https://mlsanity.com/paper/2409.01978/) (2024)

## Abstract (arXiv)

Deep learning applications require global optimization of non-convex objective functions, which have multiple local minima. The same problem is often found in physical simulations and may be resolved by the methods of Langevin dynamics with Simulated Annealing, which is a well-established approach for minimization of many-particle potentials. This analogy provides useful insights for non-convex stochastic optimization in machine learning. Here we find that integration of the discretized Langevin equation gives a coordinate updating rule equivalent to the famous Momentum optimization algorithm. As a main result, we show that a gradual decrease of the momentum coefficient from the initial value close to unity until zero is equivalent to application of Simulated Annealing or slow cooling, in physical terms. Making use of this novel approach, we propose CoolMomentum -- a new stochastic optimization method. Applying Coolmomentum to optimization of Resnet-20 on Cifar-10 dataset and Efficientnet-B0 on Imagenet, we demonstrate that it is able to achieve high accuracies.