# Decentralized Riemannian Gradient Descent on the Stiefel Manifold

arXiv: https://arxiv.org/abs/2102.07091 · HTML: https://mlsanity.com/paper/2102.07091/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 55 references that are in the map.
- First posted on arXiv 14 Feb 2021.

## Builds on (most cited first, 27 of 27)

- [Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent](https://mlsanity.com/paper/1705.09056/) (2017)
- [Unitary Evolution Recurrent Neural Networks](https://mlsanity.com/paper/1511.06464/) (2015)
- [EXTRA: An Exact First-Order Algorithm for Decentralized Consensus Optimization](https://mlsanity.com/paper/1404.6264/) (2014)
- [On the Convergence of Decentralized Gradient Descent](https://mlsanity.com/paper/1310.7063/) (2013)
- [Stochastic Gradient Push for Distributed Deep Learning](https://mlsanity.com/paper/1811.10792/) (2018)
- [Achieving Geometric Convergence for Distributed Optimization over Time-Varying Graphs](https://mlsanity.com/paper/1607.03218/) (2016)
- [Network Topology and Communication-Computation Tradeoffs in Decentralized Optimization](https://mlsanity.com/paper/1709.08765/) (2017)
- [NEXT: In-Network Nonconvex Optimization](https://mlsanity.com/paper/1602.00591/) (2016)
- [First-order Methods for Geodesically Convex Optimization](https://mlsanity.com/paper/1602.06053/) (2016)
- [Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networks](https://mlsanity.com/paper/1709.06079/) (2017)
- [On orthogonality and learning recurrent networks with long term dependencies](https://mlsanity.com/paper/1702.00071/) (2017)
- [Global rates of convergence for nonconvex optimization on manifolds](https://mlsanity.com/paper/1605.08101/) (2016)
- [On the Linear Convergence of the ADMM in Decentralized Consensus Optimization](https://mlsanity.com/paper/1307.5561/) (2013)
- [Exact Diffusion for Distributed Optimization and Learning --- Part I: Algorithm Development](https://mlsanity.com/paper/1702.05122/) (2017)
- [Multi-Agent Distributed Optimization via Inexact Consensus ADMM](https://mlsanity.com/paper/1402.6065/) (2014)
- [Decentralized Proximal Gradient Algorithms with Linear Convergence Rates](https://mlsanity.com/paper/1909.06479/) (2019)
- [DeEPCA: Decentralized Exact PCA with Linear Convergence Rate](https://mlsanity.com/paper/2102.03990/) (2021)
- [Cloud K-SVD: A Collaborative Dictionary Learning Algorithm for Big, Distributed Data](https://mlsanity.com/paper/1412.7839/) (2014)
- [Exact Diffusion for Distributed Optimization and Learning --- Part II: Convergence Analysis](https://mlsanity.com/paper/1702.05142/) (2017)
- [Distributed Linearized Alternating Direction Method of Multipliers for Composite Convex Consensus Optimization](https://mlsanity.com/paper/1512.08122/) (2015)
- [Improving the Sample and Communication Complexity for Decentralized Non-Convex Optimization: A Joint Gradient Estimation and Tracking Approach](https://mlsanity.com/paper/1910.05857/) (2019)
- [A Linearly Convergent Algorithm for Distributed Principal Component Analysis](https://mlsanity.com/paper/2101.01300/) (2021)
- [Distributed Optimization on Riemannian Manifolds for multi-agent networks](https://mlsanity.com/paper/1711.11196/) (2017)
- [High-dimensional Kuramoto models on Stiefel manifolds synchronize complex networks almost globally](https://mlsanity.com/paper/1807.10233/) (2018)
- [Weakly Convex Optimization over Stiefel Manifold Using Riemannian Subgradient-Type Methods](https://mlsanity.com/paper/1911.05047/) (2019)
- [On the Divergence of Decentralized Non-Convex Optimization](https://mlsanity.com/paper/2006.11662/) (2020)
- [On Distributed Non-convex Optimization: Projected Subgradient Method For Weakly Convex Problems in Networks](https://mlsanity.com/paper/2004.13233/) (2020)

## Built on by (most cited first, 30 of 33)

- [Personalized PCA: Decoupling Shared and Unique Features](https://mlsanity.com/paper/2207.08041/) (2022)
- [Federated Learning on Riemannian Manifolds](https://mlsanity.com/paper/2206.05668/) (2022)
- [Decentralized Optimization Over the Stiefel Manifold by an Approximate Augmented Lagrangian Function](https://mlsanity.com/paper/2112.14949/) (2021)
- [Unified Analysis of Decentralized Gradient Descent: a Contraction Mapping Framework](https://mlsanity.com/paper/2503.14353/) (2025)
- [Decentralized Riemannian Algorithm for Nonconvex Minimax Problems](https://mlsanity.com/paper/2302.03825/) (2023)
- [Decentralized Riemannian Conjugate Gradient Method on the Stiefel Manifold](https://mlsanity.com/paper/2308.10547/) (2023)
- [Nonconvex Federated Learning on Compact Smooth Submanifolds With Heterogeneous Data](https://mlsanity.com/paper/2406.08465/) (2024)
- [Stiefel Flow Matching for Moment-Constrained Structure Elucidation](https://mlsanity.com/paper/2412.12540/) (2024)
- [Analytical Survey of Learning with Low-Resource Data: From Analysis to Investigation](https://mlsanity.com/paper/2510.08962/) (2025)
- [Distributed Principal Subspace Analysis for Partitioned Big Data: Algorithms, Analysis, and Implementation](https://mlsanity.com/paper/2103.06406/) (2021)
- [Decentralized Online Riemannian Optimization Beyond Hadamard Manifolds](https://mlsanity.com/paper/2509.07779/) (2025)
- [FAST-PCA: A Fast and Exact Algorithm for Distributed Principal Component Analysis](https://mlsanity.com/paper/2108.12373/) (2021)
- [Online Optimization on Hadamard Manifolds: Curvature Independent Regret Bounds on Horospherically Convex Objectives](https://mlsanity.com/paper/2509.11236/) (2025)
- [A Variance-Reduced Stochastic Gradient Tracking Algorithm for Decentralized Optimization with Orthogonality Constraints](https://mlsanity.com/paper/2208.13643/) (2022)
- [Finite-Time Analysis of Stochastic Nonconvex Nonsmooth Optimization on the Riemannian Manifolds](https://mlsanity.com/paper/2510.21468/) (2025)
- [Exploring Diverse Generation Paths via Inference-time Stiefel Activation Steering](https://mlsanity.com/paper/2601.22010/) (2026)
- [A Survey of Learning on Small Data: Generalization, Optimization, and Challenge](https://mlsanity.com/paper/2207.14443/) (2022)
- [Decentralized Riemannian natural gradient methods with Kronecker-product approximations](https://mlsanity.com/paper/2303.09611/) (2023)
- [Decentralized Weakly Convex Optimization Over the Stiefel Manifold](https://mlsanity.com/paper/2303.17779/) (2023)
- [ADEPT: Hierarchical Bayes Approach to Personalized Federated Unsupervised Learning](https://mlsanity.com/paper/2402.12537/) (2024)
- [Decentralized Optimization on Compact Submanifolds by Quantized Riemannian Gradient Tracking](https://mlsanity.com/paper/2506.07351/) (2025)
- [Federated Learning on Riemannian Manifolds: A Gradient-Free Projection-Based Approach](https://mlsanity.com/paper/2507.22855/) (2025)
- [Decentralized Online Riemannian Optimization for Strongly Geodesically Convex Functions](https://mlsanity.com/paper/2607.20316/) (2026)
- [Decentralized Complete Dictionary Learning via ℓ4-Norm Maximization](https://mlsanity.com/paper/2211.03628/) (2022)
- [Convex optimization over a probability simplex](https://mlsanity.com/paper/2305.09046/) (2023)
- [Convergence and Complexity Guarantee for Inexact First-order Riemannian Optimization Algorithms](https://mlsanity.com/paper/2405.03073/) (2024)
- [A Double Tracking Method for Optimization with Decentralized Generalized Orthogonality Constraints](https://mlsanity.com/paper/2409.04998/) (2024)
- [Rapid Grassmannian Averaging with Chebyshev Polynomials](https://mlsanity.com/paper/2410.08956/) (2024)
- [Improved Analysis of the Accelerated Noisy Power Method with Applications to Decentralized PCA](https://mlsanity.com/paper/2602.03682/) (2026)
- [Riemannian Stochastic Optimization for Sufficient Dimension Reduction](https://mlsanity.com/paper/2606.00413/) (2026)

## Abstract (arXiv)

We consider a distributed non-convex optimization where a network of agents aims at minimizing a global function over the Stiefel manifold. The global function is represented as a finite sum of smooth local functions, where each local function is associated with one agent and agents communicate with each other over an undirected connected graph. The problem is non-convex as local functions are possibly non-convex (but smooth) and the Steifel manifold is a non-convex set. We present a decentralized Riemannian stochastic gradient method (DRSGD) with the convergence rate of O(1/√K) to a stationary point. To have exact convergence with constant stepsize, we also propose a decentralized Riemannian gradient tracking algorithm (DRGTA) with the convergence rate of O(1/K) to a stationary point. We use multi-step consensus to preserve the iteration in the local (consensus) region. DRGTA is the first decentralized algorithm with exact convergence for distributed optimization on Stiefel manifold.