# An Algebraically Converging Stochastic Gradient Descent Algorithm for Global Optimization

arXiv: https://arxiv.org/abs/2204.05923 · HTML: https://mlsanity.com/paper/2204.05923/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 29 references that are in the map.
- First posted on arXiv 12 Apr 2022.

## Builds on (most cited first, 6 of 6)

- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Gradient Descent with Random Initialization: Fast Global Convergence for Nonconvex Phase Retrieval](https://mlsanity.com/paper/1803.07726/) (2018)
- [A consensus-based global optimization method for high dimensional machine learning problems](https://mlsanity.com/paper/1909.09249/) (2019)
- [State-Dependent Temperature Control for Langevin Diffusions](https://mlsanity.com/paper/2011.07456/) (2020)
- [Replica Exchange for Non-Convex Optimization](https://mlsanity.com/paper/2001.08356/) (2020)
- [Trends in Consensus-based optimization](https://mlsanity.com/paper/2104.01383/) (2021)

## Built on by (most cited first, 3 of 3)

- [Adaptive State-Dependent Diffusion for Derivative-Free Optimization](https://mlsanity.com/paper/2302.04370/) (2023)
- [Sampling with Adaptive Variance for Multimodal Distributions](https://mlsanity.com/paper/2411.15220/) (2024)
- [Accelerating Convergence in Global Non-Convex Optimization with Reversible Diffusion](https://mlsanity.com/paper/2305.11493/) (2023)

## Abstract (arXiv)

We propose a new gradient descent algorithm with added stochastic terms for finding the global optimizers of nonconvex optimization problems. A key component in the algorithm is the adaptive tuning of the randomness based on the value of the objective function. In the language of simulated annealing, the temperature is state-dependent. With this, we prove the global convergence of the algorithm with an algebraic rate both in probability and in the parameter space. This is a significant improvement over the classical rate from using a more straightforward control of the noise term. The convergence proof is based on the actual discrete setup of the algorithm, not just its continuous limit as often done in the literature. We also present several numerical examples to demonstrate the efficiency and robustness of the algorithm for reasonably complex objective functions.