# Averaging Stochastic Gradient Descent on Riemannian Manifolds

arXiv: https://arxiv.org/abs/1802.09128 · HTML: https://mlsanity.com/paper/1802.09128/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Feb 2018.

## Built on by (most cited first, 30 of 39)

- [Fast and accurate optimization on the orthogonal manifold without retraction](https://mlsanity.com/paper/2102.07432/) (2021)
- [Convergence Rates of Stochastic Gradient Descent under Infinite Noise Variance](https://mlsanity.com/paper/2102.10346/) (2021)
- [NRDF: Neural Riemannian Distance Fields for Learning Articulated Pose Priors](https://mlsanity.com/paper/2403.03122/) (2024)
- [Designing Universal Causal Deep Learning Models: The Geometric (Hyper)Transformer](https://mlsanity.com/paper/2201.13094/) (2022)
- [Provably Optimal Memory Capacity for Modern Hopfield Models: Transformer-Compatible Dense Associative Memories as Spherical Codes](https://mlsanity.com/paper/2410.23126/) (2024)
- [Infeasible Deterministic, Stochastic, and Variance-Reduction Algorithms for Optimization under Orthogonality Constraints](https://mlsanity.com/paper/2303.16510/) (2023)
- [Optimal variance-reduced stochastic approximation in Banach spaces](https://mlsanity.com/paper/2201.08518/) (2022)
- [On Riemannian Stochastic Approximation Schemes with Fixed Step-Size](https://mlsanity.com/paper/2102.07586/) (2021)
- [First-Order Algorithms for Min-Max Optimization in Geodesic Metric Spaces](https://mlsanity.com/paper/2206.02041/) (2022)
- [Sion's Minimax Theorem in Geodesic Metric Spaces and a Riemannian Extragradient Algorithm](https://mlsanity.com/paper/2202.06950/) (2022)
- [Learning Rate Schedules in the Presence of Distribution Shift](https://mlsanity.com/paper/2303.15634/) (2023)
- [Learning Optimal Power Flow Value Functions with Input-Convex Neural Networks](https://mlsanity.com/paper/2310.04605/) (2023)
- [Riemannian accelerated gradient methods via extrapolation](https://mlsanity.com/paper/2208.06619/) (2022)
- [Riemannian Optimization on Relaxed Indicator Matrix Manifold](https://mlsanity.com/paper/2503.20505/) (2025)
- [No-go Theorem for Acceleration in the Hyperbolic Plane](https://mlsanity.com/paper/2101.05657/) (2021)
- [Accelerated Riemannian Optimization: Handling Constraints with a Prox to Bound Geometric Penalties](https://mlsanity.com/paper/2211.14645/) (2022)
- [Fairness Uncertainty Quantification: How certain are you that the model is fair?](https://mlsanity.com/paper/2304.13950/) (2023)
- [Riemannian coordinate descent algorithms on matrix manifolds](https://mlsanity.com/paper/2406.02225/) (2024)
- [Riemannian Federated Learning via Averaging Gradient Streams](https://mlsanity.com/paper/2409.07223/) (2024)
- [A quantitative Robbins-Siegmund theorem](https://mlsanity.com/paper/2410.15986/) (2024)
- [Riemannian stochastic approximation algorithms](https://mlsanity.com/paper/2206.06795/) (2022)
- [Convergence of variational Monte Carlo simulation and scale-invariant pre-training](https://mlsanity.com/paper/2303.11602/) (2023)
- [Low-complexity subspace-descent over symmetric positive definite manifold](https://mlsanity.com/paper/2305.02041/) (2023)
- [Last-Iterate Convergence of Adaptive Riemannian Gradient Descent for Equilibrium Computation](https://mlsanity.com/paper/2306.16617/) (2023)
- [Stochastic Modified Flows for Riemannian Stochastic Gradient Descent](https://mlsanity.com/paper/2402.03467/) (2024)
- [Streamlining in the Riemannian Realm: Efficient Riemannian Optimization with Loopless Variance Reduction](https://mlsanity.com/paper/2403.06677/) (2024)
- [Faster Convergence of Riemannian Stochastic Gradient Descent with Increasing Batch Size](https://mlsanity.com/paper/2501.18164/) (2025)
- [A Geometric Analysis of PCA](https://mlsanity.com/paper/2510.20978/) (2025)
- [Towards Modeling and Resolving Singular Parameter Spaces using Stratifolds](https://mlsanity.com/paper/2112.03734/) (2021)
- [Riemannian Stochastic Gradient Method for Nested Composition Optimization](https://mlsanity.com/paper/2207.09350/) (2022)

## Abstract (arXiv)

We consider the minimization of a function defined on a Riemannian manifold M accessible only through unbiased estimates of its gradients. We develop a geometric framework to transform a sequence of slowly converging iterates generated from stochastic gradient descent (SGD) on M to an averaged iterate sequence with a robust and fast O(1/n) convergence rate. We then present an application of our framework to geodesically-strongly-convex (and possibly Euclidean non-convex) problems. Finally, we demonstrate how these ideas apply to the case of streaming k-PCA, where we show how to accelerate the slow rate of the randomized power method (without requiring knowledge of the eigengap) into a robust algorithm achieving the optimal rate of convergence.