# A Continuous-time Stochastic Gradient Descent Method for Continuous Data

arXiv: https://arxiv.org/abs/2112.03754 · HTML: https://mlsanity.com/paper/2112.03754/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 58 references that are in the map.
- First posted on arXiv 7 Dec 2021.

## Builds on (most cited first, 20 of 20)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [Certified Adversarial Robustness via Randomized Smoothing](https://mlsanity.com/paper/1902.02918/) (2019)
- [DGM: A deep learning algorithm for solving partial differential equations](https://mlsanity.com/paper/1708.07469/) (2017)
- [DeepXDE: A deep learning library for solving differential equations](https://mlsanity.com/paper/1907.04502/) (2019)
- [Convex Optimization: Algorithms and Complexity](https://mlsanity.com/paper/1405.4980/) (2014)
- [Robust Adversarial Reinforcement Learning](https://mlsanity.com/paper/1703.02702/) (2017)
- [SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives](https://mlsanity.com/paper/1407.0202/) (2014)
- [Stochastic Gradient Descent as Approximate Bayesian Inference](https://mlsanity.com/paper/1704.04289/) (2017)
- [On the Origin of Implicit Regularization in Stochastic Gradient Descent](https://mlsanity.com/paper/2101.12176/) (2021)
- [Stochastic modified equations and adaptive stochastic gradient algorithms](https://mlsanity.com/paper/1511.06251/) (2015)
- [Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundations](https://mlsanity.com/paper/1811.01558/) (2018)
- [The Implicit Regularization of Stochastic Gradient Flow for Least Squares](https://mlsanity.com/paper/2003.07802/) (2020)
- [A Variational Analysis of Stochastic Gradient Algorithms](https://mlsanity.com/paper/1602.02666/) (2016)
- [Continuous Time Analysis of Momentum Methods](https://mlsanity.com/paper/1906.04285/) (2019)
- [Stochastic Primal-Dual Hybrid Gradient Algorithm with Arbitrary Sampling and Imaging Applications](https://mlsanity.com/paper/1706.04957/) (2017)
- [Stochastic gradient descent with noise of machine learning type. Part II: Continuous time analysis](https://mlsanity.com/paper/2106.02588/) (2021)
- [Analysis of Stochastic Gradient Descent in Continuous Time](https://mlsanity.com/paper/2004.07177/) (2020)
- [Stochastic Gradient Descent in Continuous Time](https://mlsanity.com/paper/1611.05545/) (2016)
- [Explicit Stabilised Gradient Descent for Faster Strongly Convex Optimisation](https://mlsanity.com/paper/1805.07199/) (2018)

## Built on by (most cited first, 7 of 7)

- [Can Physics-Informed Neural Networks beat the Finite Element Method?](https://mlsanity.com/paper/2302.04107/) (2023)
- [A Forward Propagation Algorithm for Online Optimization of Nonlinear Stochastic Differential Equations](https://mlsanity.com/paper/2207.04496/) (2022)
- [Losing momentum in continuous-time stochastic optimisation](https://mlsanity.com/paper/2209.03705/) (2022)
- [Gradient flows and randomised thresholding: sparse inversion and classification](https://mlsanity.com/paper/2203.11555/) (2022)
- [Subsampling Error in Stochastic Gradient Langevin Diffusions](https://mlsanity.com/paper/2305.13882/) (2023)
- [How to beat a Bayesian adversary](https://mlsanity.com/paper/2407.08678/) (2024)
- [Convergence of the generalization error for deep gradient flow methods for PDEs](https://mlsanity.com/paper/2512.25017/) (2025)

## Abstract (arXiv)

Optimization problems with continuous data appear in, e.g., robust machine learning, functional data analysis, and variational inference. Here, the target function is given as an integral over a family of (continuously) indexed target functions - integrated with respect to a probability measure. Such problems can often be solved by stochastic optimization methods: performing optimization steps with respect to the indexed target function with randomly switched indices. In this work, we study a continuous-time variant of the stochastic gradient descent algorithm for optimization problems with continuous data. This so-called stochastic gradient process consists in a gradient flow minimizing an indexed target function that is coupled with a continuous-time index process determining the index. Index processes are, e.g., reflected diffusions, pure jump processes, or other L'evy processes on compact spaces. Thus, we study multiple sampling patterns for the continuous data space and allow for data simulated or streamed at runtime of the algorithm. We analyze the approximation properties of the stochastic gradient process and study its longtime behavior and ergodicity under constant and decreasing learning rates. We end with illustrating the applicability of the stochastic gradient process in a polynomial regression problem with noisy functional data, as well as in a physics-informed neural network.