# Analysis of Stochastic Gradient Descent in Continuous Time

arXiv: https://arxiv.org/abs/2004.07177 · HTML: https://mlsanity.com/paper/2004.07177/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Apr 2020.

## Built on by (most cited first, 24 of 24)

- [Stochastic gradient descent with noise of machine learning type. Part II: Continuous time analysis](https://mlsanity.com/paper/2106.02588/) (2021)
- [A Continuized View on Nesterov Acceleration for Stochastic Gradient Descent and Randomized Gossip](https://mlsanity.com/paper/2106.07644/) (2021)
- [Stationary Behavior of Constant Stepsize SGD Type Algorithms: An Asymptotic Characterization](https://mlsanity.com/paper/2111.06328/) (2021)
- [DADAO: Decoupled Accelerated Decentralized Asynchronous Optimization](https://mlsanity.com/paper/2208.00779/) (2022)
- [A Continuous-time Stochastic Gradient Descent Method for Continuous Data](https://mlsanity.com/paper/2112.03754/) (2021)
- [Noise Balance and Stationary Distribution of Stochastic Gradient Descent](https://mlsanity.com/paper/2308.06671/) (2023)
- [Backward error analysis and the qualitative behaviour of stochastic optimization algorithms: Application to stochastic coordinate descent](https://mlsanity.com/paper/2309.02082/) (2023)
- [Losing momentum in continuous-time stochastic optimisation](https://mlsanity.com/paper/2209.03705/) (2022)
- [NAG-GS: Semi-Implicit, Accelerated and Robust Stochastic Optimizer](https://mlsanity.com/paper/2209.14937/) (2022)
- [Gradient flows and randomised thresholding: sparse inversion and classification](https://mlsanity.com/paper/2203.11555/) (2022)
- [A PDE-based Explanation of Extreme Numerical Sensitivities and Edge of Stability in Training Neural Networks](https://mlsanity.com/paper/2206.02001/) (2022)
- [Beyond first-order methods for non-convex non-concave min-max optimization](https://mlsanity.com/paper/2304.08389/) (2023)
- [Subsampling Error in Stochastic Gradient Langevin Diffusions](https://mlsanity.com/paper/2305.13882/) (2023)
- [The Expected Loss of Preconditioned Langevin Dynamics Reveals the Hessian Rank](https://mlsanity.com/paper/2402.13810/) (2024)
- [How to beat a Bayesian adversary](https://mlsanity.com/paper/2407.08678/) (2024)
- [Learning Gradient Flow: Using Equation Discovery to Accelerate Engineering Optimization](https://mlsanity.com/paper/2602.13513/) (2026)
- [Analysis of Kinetic Models for Label Switching and Stochastic Gradient Descent](https://mlsanity.com/paper/2207.00389/) (2022)
- [Online Learning for the Random Feature Model in the Student-Teacher Framework](https://mlsanity.com/paper/2303.14083/) (2023)
- [Doubly Stochastic Models: Learning with Unbiased Label Noises and Inference Stability](https://mlsanity.com/paper/2304.00320/) (2023)
- [Effect of Random Learning Rate: Theoretical Analysis of SGD Dynamics in Non-Convex Optimization via Stationary Distribution](https://mlsanity.com/paper/2406.16032/) (2024)
- [Gradient-Free Sequential Bayesian Experimental Design via Interacting Particle Systems](https://mlsanity.com/paper/2504.13320/) (2025)
- [Heavy-Ball Momentum Method in Continuous Time and Discretization Error Analysis](https://mlsanity.com/paper/2506.14806/) (2025)
- [Convergence of the generalization error for deep gradient flow methods for PDEs](https://mlsanity.com/paper/2512.25017/) (2025)
- [Dimensionless Controls of Plasticity Under Alternating Tasks: From Evolutionary Biology to Continual Learning](https://mlsanity.com/paper/2608.23889/) (2026)

## Abstract (arXiv)

Stochastic gradient descent is an optimisation method that combines classical gradient descent with random subsampling within the target functional. In this work, we introduce the stochastic gradient process as a continuous-time representation of stochastic gradient descent. The stochastic gradient process is a dynamical system that is coupled with a continuous-time Markov process living on a finite state space. The dynamical system -- a gradient flow -- represents the gradient descent part, the process on the finite state space represents the random subsampling. Processes of this type are, for instance, used to model clonal populations in fluctuating environments. After introducing it, we study theoretical properties of the stochastic gradient process: We show that it converges weakly to the gradient flow with respect to the full target function, as the learning rate approaches zero. We give conditions under which the stochastic gradient process with constant learning rate is exponentially ergodic in the Wasserstein sense. Then we study the case, where the learning rate goes to zero sufficiently slowly and the single target functions are strongly convex. In this case, the process converges weakly to the point mass concentrated in the global minimum of the full target function; indicating consistency of the method. We conclude after a discussion of discretisation strategies for the stochastic gradient process and numerical experiments.