mlsanityData to 7 Oct 2026

Continuous-time Riemannian SGD and SVRG Flows on Wasserstein Probabilistic Space

arXiv 2401.13530 ↗First posted 24 Jan 2024cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Recently, optimization on the Riemannian manifold have provided valuable insights to the optimization community. In this regard, extending these methods to to the Wasserstein space is of particular interest, since optimization on Wasserstein space is closely connected to practical sampling processes. Generally, the standard (continuous) optimization method on Wasserstein space is Riemannian gradient flow (i.e., Langevin dynamics when minimizing KL divergence). In this paper, we aim to enrich the family of continuous optimization methods in the Wasserstein space, by extending the gradient flow on it into the stochastic gradient descent (SGD) flow and stochastic variance reduction gradient (SVRG) flow. By leveraging the property of Wasserstein space, we construct stochastic differential equations (SDEs) to approximate the corresponding discrete Euclidean dynamics of the desired Riemannian stochastic methods. Then, we obtain the flows in Wasserstein space by Fokker-Planck equation. Finally, we establish convergence rates of the proposed stochastic flows, which align with those known in the Euclidean setting.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 33 of its 59 references that are in the map. First posted on arXiv 24 Jan 2024.

Builds on

33 of 59 references.

References · 33

  1. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  2. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  3. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 716
  4. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  5. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 526
  6. 2018SPIDER: Near-Optimal Non-Convex Optimization via Stochastic Path Integrated Differential Estimatorcited by 283
  7. 2017SARAH: A Novel Method for Machine Learning Problems Using Stochastic Recursive Gradientcited by 218
  8. 2015A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insightscited by 203
  9. 2018Riemannian Adaptive Optimization Methodscited by 187
  10. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 172
  11. 2013Minimizing Finite Sums with the Stochastic Average Gradientcited by 164
  12. 2017Stein Variational Gradient Descent as Gradient Flowcited by 149
21 more
  1. 2016High-dimensional Bayesian inference via the Unadjusted Langevin Algorithmcited by 144
  2. 2016Stochastic Variance Reduction for Nonconvex Optimizationcited by 143
  3. 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 141
  4. 2018Log-concave sampling: Metropolis-Hastings algorithms are fastcited by 117
  5. 2016First-order Methods for Geodesically Convex Optimizationcited by 92
  6. 2020Interacting particle solutions of Fokker-Planck equations through gradient-log-density estimationcited by 87
  7. 2017Convergence of Langevin MCMC in KL-divergencecited by 86
  8. 2016Riemannian SVRG: Fast Stochastic Optimization on Riemannian Manifoldscited by 64
  9. 2022Towards a Theory of Non-Log-Concave Sampling: First-Order Stationarity Guarantees for Langevin Monte Carlocited by 63
  10. 2016Global rates of convergence for nonconvex optimization on manifoldscited by 61
  11. 2019Accelerated Information Gradient flowcited by 38
  12. 2017Riemannian approach to batch normalizationcited by 35
  13. 2023Towards a Complete Analysis of Langevin Monte Carlo: Beyond Poincar'e Inequalitycited by 31
  14. 2020Information Newton's flow: second-order optimization method in probability spacecited by 22
  15. 2018On the Theory of Variance Reduction for Stochastic Gradient Monte Carlocited by 19
  16. 2018Continuous-time Models for Stochastic Optimization Algorithmscited by 16
  17. 2022Improved Convergence Rate of Stochastic Gradient Langevin Dynamics with Variance Reduction and its Application to Optimizationcited by 16
  18. 2018Towards Riemannian Accelerated Gradient Methodscited by 12
  19. 2020Characterization of Excess Risk for Locally Strongly Convex Population Riskcited by 5
  20. 2018R-SPIDER: A Fast Riemannian Stochastic Optimization Algorithm with Curvature Independent Ratecited by 4
  21. 2021Accelerating Training of Batch Normalization: A Manifold Perspectivecited by 4

Built on by

Not cited yet.