# Bayesian interpretation of SGD as Ito process

arXiv: https://arxiv.org/abs/1911.09011 · HTML: https://mlsanity.com/paper/1911.09011/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Nov 2019.

## Built on by (most cited first, 3 of 3)

- [Do Parameters Reveal More than Loss for Membership Inference?](https://mlsanity.com/paper/2406.11544/) (2024)
- [Hamiltonian Monte Carlo Particle Swarm Optimizer](https://mlsanity.com/paper/2206.14134/) (2022)
- [Variational Stochastic Gradient Descent for Deep Neural Networks](https://mlsanity.com/paper/2404.06549/) (2024)

## Abstract (arXiv)

The current interpretation of stochastic gradient descent (SGD) as a stochastic process lacks generality in that its numerical scheme restricts continuous-time dynamics as well as the loss function and the distribution of gradient noise. We introduce a simplified scheme with milder conditions that flexibly interprets SGD as a discrete-time approximation of an Ito process. The scheme also works as a common foundation of SGD and stochastic gradient Langevin dynamics (SGLD), providing insights into their asymptotic properties. We investigate the convergence of SGD with biased gradient in terms of the equilibrium mode and the overestimation problem of the second moment of SGLD.