# Stochastic Runge-Kutta methods and adaptive SGD-G2 stochastic gradient descent

arXiv: https://arxiv.org/abs/2002.09304 · HTML: https://mlsanity.com/paper/2002.09304/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Feb 2020.

## Built on by (most cited first, 4 of 4)

- [The convergence of the Stochastic Gradient Descent (SGD) : a self-contained proof](https://mlsanity.com/paper/2103.14350/) (2021)
- [Unbiased and Sign Compression in Distributed Learning: Comparing Noise Resilience via SDEs](https://mlsanity.com/paper/2502.17009/) (2025)
- [Towards Hyperparameter-Agnostic DNN Training via Dynamical System Insights](https://mlsanity.com/paper/2310.13901/) (2023)
- [Learning by solving differential equations](https://mlsanity.com/paper/2505.13397/) (2025)

## Abstract (arXiv)

The minimization of the loss function is of paramount importance in deep neural networks. On the other hand, many popular optimization algorithms have been shown to correspond to some evolution equation of gradient flow type. Inspired by the numerical schemes used for general evolution equations we introduce a second order stochastic Runge Kutta method and show that it yields a consistent procedure for the minimization of the loss function. In addition it can be coupled, in an adaptive framework, with a Stochastic Gradient Descent (SGD) to adjust automatically the learning rate of the SGD, without the need of any additional information on the Hessian of the loss functional. The adaptive SGD, called SGD-G2, is successfully tested on standard datasets.