# Uniform-in-Time Weak Error Analysis for Stochastic Gradient Descent Algorithms via Diffusion Approximation

arXiv: https://arxiv.org/abs/1902.00635 · HTML: https://mlsanity.com/paper/1902.00635/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Feb 2019.

## Built on by (most cited first, 15 of 15)

- [Toward Equation of Motion for Deep Neural Networks: Continuous-time Gradient Descent and Discretization Error Analysis](https://mlsanity.com/paper/2210.15898/) (2022)
- [Stochastic Modified Flows, Mean-Field Limits and Dynamics of Stochastic Gradient Descent](https://mlsanity.com/paper/2302.07125/) (2023)
- [On uniform-in-time diffusion approximation for stochastic gradient descent](https://mlsanity.com/paper/2207.04922/) (2022)
- [Topological obstruction to the training of shallow ReLU neural networks](https://mlsanity.com/paper/2410.14837/) (2024)
- [Topology and Geometry of the Learning Space of ReLU Networks: Connectivity and Singularities](https://mlsanity.com/paper/2602.00693/) (2026)
- [A sharp uniform-in-time error estimate for Stochastic Gradient Langevin Dynamics](https://mlsanity.com/paper/2207.09304/) (2022)
- [Stochastic Modified Flows for Riemannian Stochastic Gradient Descent](https://mlsanity.com/paper/2402.03467/) (2024)
- [On Interpolation Formulas Describing Neural Network Generalization](https://mlsanity.com/paper/2603.13872/) (2026)
- [Towards Universal Convergence of Backward Error in Linear System Solvers](https://mlsanity.com/paper/2604.16075/) (2026)
- [Analysis of Kinetic Models for Label Switching and Stochastic Gradient Descent](https://mlsanity.com/paper/2207.00389/) (2022)
- [Ito Diffusion Approximation of Universal Ito Chains for Sampling, Optimization and Boosting](https://mlsanity.com/paper/2310.06081/) (2023)
- [A General Continuous-Time Formulation of Stochastic ADMM and Its Variants](https://mlsanity.com/paper/2404.14358/) (2024)
- [Do Deep Networks Forget Initialization? A Forgetting-Time View of Practical Inductive Bias](https://mlsanity.com/paper/2605.29152/) (2026)
- [Second-Order Path Kernel Interpolation Formulas in Machine Learning](https://mlsanity.com/paper/2606.07495/) (2026)
- [A Rate Transfer Method: Convergence Rates in Stochastic Gradient Descent via Stochastic Calculus](https://mlsanity.com/paper/2610.18739/) (2026)

## Abstract (arXiv)

Diffusion approximation provides weak approximation for stochastic gradient descent algorithms in a finite time horizon. In this paper, we introduce new tools motivated by the backward error analysis of numerical stochastic differential equations into the theoretical framework of diffusion approximation, extending the validity of the weak approximation from finite to infinite time horizon. The new techniques developed in this paper enable us to characterize the asymptotic behavior of constant-step-size SGD algorithms for strongly convex objective functions, a goal previously unreachable within the diffusion approximation framework. Our analysis builds upon a truncated formal power expansion of the solution of a stochastic modified equation arising from diffusion approximation, where the main technical ingredient is a uniform-in-time weak error bound controlling the long-term behavior of the expansion coefficient functions near the global minimum. We expect these new techniques to greatly expand the range of applicability of diffusion approximation to cover wider and deeper aspects of stochastic optimization algorithms in data science.