# Riemannian coordinate descent algorithms on matrix manifolds

arXiv: https://arxiv.org/abs/2406.02225 · HTML: https://mlsanity.com/paper/2406.02225/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 75 references that are in the map.
- First posted on arXiv 4 Jun 2024.

## Builds on (most cited first, 30 of 41)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Poincar'e Embeddings for Learning Hierarchical Representations](https://mlsanity.com/paper/1705.08039/) (2017)
- [Unitary Evolution Recurrent Neural Networks](https://mlsanity.com/paper/1511.06464/) (2015)
- [Learning Continuous Hierarchies in the Lorentz Model of Hyperbolic Geometry](https://mlsanity.com/paper/1806.03417/) (2018)
- [Coordinate Descent Algorithms](https://mlsanity.com/paper/1502.04759/) (2015)
- [Orthogonal Convolutional Neural Networks](https://mlsanity.com/paper/1911.12207/) (2019)
- [Manopt, a Matlab toolbox for optimization on manifolds](https://mlsanity.com/paper/1308.5200/) (2013)
- [Trivializations for Gradient-Based Optimization on Manifolds](https://mlsanity.com/paper/1909.09501/) (2019)
- [Riemannian SVRG: Fast Stochastic Optimization on Riemannian Manifolds](https://mlsanity.com/paper/1605.07147/) (2016)
- [Averaging Stochastic Gradient Descent on Riemannian Manifolds](https://mlsanity.com/paper/1802.09128/) (2018)
- [Fast and accurate optimization on the orthogonal manifold without retraction](https://mlsanity.com/paper/2102.07432/) (2021)
- [On Riemannian Optimization over Positive Definite Matrices with the Bures-Wasserstein Geometry](https://mlsanity.com/paper/2106.00286/) (2021)
- [A Riemannian Block Coordinate Descent Method for Computing the Projection Robust Wasserstein Distance](https://mlsanity.com/paper/2012.05199/) (2020)
- [Learning Multilingual Word Embeddings in Latent Metric Space: A Geometric Approach](https://mlsanity.com/paper/1808.08773/) (2018)
- [Infeasible Deterministic, Stochastic, and Variance-Reduction Algorithms for Optimization under Orthogonality Constraints](https://mlsanity.com/paper/2303.16510/) (2023)
- [Parallelizable Algorithms for Optimization Problems with Orthogonality Constraints](https://mlsanity.com/paper/1810.03930/) (2018)
- [Differential Privacy Over Riemannian Manifolds](https://mlsanity.com/paper/2111.02516/) (2021)
- [Block Coordinate Descent on Smooth Manifolds: Convergence Theory and Twenty-One Examples](https://mlsanity.com/paper/2305.14744/) (2023)
- [Geometry-aware Domain Adaptation for Unsupervised Alignment of Word Embeddings](https://mlsanity.com/paper/2004.08243/) (2020)
- [Federated Learning on Riemannian Manifolds](https://mlsanity.com/paper/2206.05668/) (2022)
- [Manifold Optimization Over the Set of Doubly Stochastic Matrices: A Second-Order Geometry](https://mlsanity.com/paper/1802.02628/) (2018)
- [Dissolving Constraints for Riemannian Optimization](https://mlsanity.com/paper/2203.10319/) (2022)
- [Riemannian Hamiltonian methods for min-max optimization on manifolds](https://mlsanity.com/paper/2204.11418/) (2022)
- [Differentially private Riemannian optimization](https://mlsanity.com/paper/2205.09494/) (2022)
- [A dual framework for low-rank tensor completion](https://mlsanity.com/paper/1712.01193/) (2017)
- [Sion's Minimax Theorem in Geodesic Metric Spaces and a Riemannian Extragradient Algorithm](https://mlsanity.com/paper/2202.06950/) (2022)
- [Coordinate Descent Without Coordinates: Tangent Subspace Descent on Riemannian Manifolds](https://mlsanity.com/paper/1912.10627/) (2019)
- [Riemannian Optimization on the Symplectic Stiefel Manifold](https://mlsanity.com/paper/2006.15226/) (2020)
- [Manifold optimization for non-linear optimal transport problems](https://mlsanity.com/paper/2103.00902/) (2021)
- [Riemannian block SPD coupling manifold and its application to optimal transport](https://mlsanity.com/paper/2201.12933/) (2022)

## Built on by (most cited first, 4 of 4)

- [Lie Algebra Canonicalization: Equivariant Neural Operators under arbitrary Lie Groups](https://mlsanity.com/paper/2410.02698/) (2024)
- [Efficient Optimization with Orthogonality Constraint: a Randomized Riemannian Submanifold Method](https://mlsanity.com/paper/2505.12378/) (2025)
- [An Embarrassingly Simple Way to Optimize Orthogonal Matrices at Scale](https://mlsanity.com/paper/2602.14656/) (2026)
- [Differentially Private Geodesic Regression](https://mlsanity.com/paper/2504.11304/) (2025)

## Abstract (arXiv)

Many machine learning applications are naturally formulated as optimization problems on Riemannian manifolds. The main idea behind Riemannian optimization is to maintain the feasibility of the variables while moving along a descent direction on the manifold. This results in updating all the variables at every iteration. In this work, we provide a general framework for developing computationally efficient coordinate descent (CD) algorithms on matrix manifolds that allows updating only a few variables at every iteration while adhering to the manifold constraint. In particular, we propose CD algorithms for various manifolds such as Stiefel, Grassmann, (generalized) hyperbolic, symplectic, and symmetric positive (semi)definite. While the cost per iteration of the proposed CD algorithms is low, we further develop a more efficient variant via a first-order approximation of the objective function. We analyze their convergence and complexity, and empirically illustrate their efficacy in several applications.