Stochastic Modified Flows for Riemannian Stochastic Gradient Descent
Not on the map. Why?
Abstract (arXiv)
We give quantitative estimates for the rate of convergence of Riemannian stochastic gradient descent (RSGD) to Riemannian gradient flow and to a diffusion process, the so-called Riemannian stochastic modified flow (RSMF). Using tools from stochastic differential geometry we show that, in the small learning rate regime, RSGD can be approximated by the solution to the RSMF driven by an infinite-dimensional Wiener process. The RSMF accounts for the random fluctuations of RSGD and, thereby, increases the order of approximation compared to the deterministic Riemannian gradient flow. The RSGD is build using the concept of a retraction map, that is, a cost efficient approximation of the exponential map, and we prove quantitative bounds for the weak error of the diffusion approximation under assumptions on the retraction map, the geometry of the manifold, and the random estimators of the gradient.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 27 of its 64 references that are in the map. First posted on arXiv 2 Feb 2024.
Builds on
27 of 64 references.
References · 27
- 2017Poincar'e Embeddings for Learning Hierarchical Representationscited by 674
- 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
- 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 141
- 2018Improving GANs Using Optimal Transportcited by 122
- 2018Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundationscited by 109
- 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 106
- 2021On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)cited by 73
- 2019Convergence rates for the stochastic gradient descent method for non-convex objective functionscited by 66
- 2016Riemannian SVRG: Fast Stochastic Optimization on Riemannian Manifoldscited by 64
- 2017Neural Embeddings of Graphs in Hyperbolic Spacecited by 58
- 2020Projection Robust Wasserstein Distance and Riemannian Optimizationcited by 45
- 2018Averaging Stochastic Gradient Descent on Riemannian Manifoldscited by 39
15 more
- 2019Escaping from saddle points on Riemannian manifoldscited by 33
- 2021Stochastic gradient descent with noise of machine learning type. Part II: Continuous time analysiscited by 25
- 2019Uniform-in-Time Weak Error Analysis for Stochastic Gradient Descent Algorithms via Diffusion Approximationcited by 14
- 2021On minimal representations of shallow ReLU networkscited by 13
- 2021On Riemannian Stochastic Approximation Schemes with Fixed Step-Sizecited by 12
- 2018Gradient descent in hyperbolic spacecited by 11
- 2020Sinkhorn Natural Gradient for Generative Modelscited by 10
- 2023Robust Implicit Regularization via Weight Normalizationcited by 10
- 2023Riemannian stochastic optimization methods avoid strict saddle pointscited by 8
- 2017Stochastic Approximation on Riemannian manifoldscited by 7
- 2020Convergence Analysis of Riemannian Stochastic Approximation Schemescited by 5
- 2020Inductive Bias of Gradient Descent for Weight Normalized Smooth Homogeneous Neural Netscited by 5
- 2023Unlocking optimal batch size schedules using continuous-time control and perturbation theorycited by 3
- 2023Convergence of Riemannian Stochastic Gradient Descent on Hadamard Manifoldcited by 3
- 2022Riemannian stochastic approximation algorithmscited by 2
Built on by
Citing papers per year: 2025: 2.
2 papers.