# Optimization without Retraction on the Random Generalized Stiefel Manifold

arXiv: https://arxiv.org/abs/2405.01702 · HTML: https://mlsanity.com/paper/2405.01702/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 48 references that are in the map.
- First posted on arXiv 2 May 2024.

## Builds on (most cited first, 23 of 23)

- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [First-order Methods for Geodesically Convex Optimization](https://mlsanity.com/paper/1602.06053/) (2016)
- [Riemannian SVRG: Fast Stochastic Optimization on Riemannian Manifolds](https://mlsanity.com/paper/1605.07147/) (2016)
- [Global rates of convergence for nonconvex optimization on manifolds](https://mlsanity.com/paper/1605.08101/) (2016)
- [Fast and accurate optimization on the orthogonal manifold without retraction](https://mlsanity.com/paper/2102.07432/) (2021)
- [From Nesterov's Estimate Sequence to Riemannian Acceleration](https://mlsanity.com/paper/2001.08876/) (2020)
- [Riemannian preconditioning](https://mlsanity.com/paper/1405.6055/) (2014)
- [Infeasible Deterministic, Stochastic, and Variance-Reduction Algorithms for Optimization under Orthogonality Constraints](https://mlsanity.com/paper/2303.16510/) (2023)
- [Parallelizable Algorithms for Optimization Problems with Orthogonality Constraints](https://mlsanity.com/paper/1810.03930/) (2018)
- [Orthogonal Directions Constrained Gradient Method: from non-linear equality constraints to Stiefel manifold](https://mlsanity.com/paper/2303.09261/) (2023)
- [Dissolving Constraints for Riemannian Optimization](https://mlsanity.com/paper/2203.10319/) (2022)
- [Efficient Algorithms for Large-scale Generalized Eigenvector Computation and Canonical Correlation Analysis](https://mlsanity.com/paper/1604.03930/) (2016)
- [Optimization flows landing on the Stiefel manifold](https://mlsanity.com/paper/2202.09058/) (2022)
- [Faster ICA under orthogonal constraint](https://mlsanity.com/paper/1711.10873/) (2017)
- [Stochastic Canonical Correlation Analysis](https://mlsanity.com/paper/1702.06533/) (2017)
- [Stochastic Approximation for Canonical Correlation Analysis](https://mlsanity.com/paper/1702.06818/) (2017)
- [An Online Riemannian PCA for Stochastic Canonical Correlation Analysis](https://mlsanity.com/paper/2106.07479/) (2021)
- [Computing second-order points under equality constraints: revisiting Fletcher's augmented Lagrangian](https://mlsanity.com/paper/2204.01448/) (2022)
- [Doubly Accelerated Methods for Faster CCA and Generalized Eigendecomposition](https://mlsanity.com/paper/1607.06017/) (2016)
- [Finding Linear Structure in Large Datasets with Scalable Canonical Correlation Analysis](https://mlsanity.com/paper/1506.08170/) (2015)
- [Efficient Globally Convergent Stochastic Optimization for Canonical Correlation Analysis](https://mlsanity.com/paper/1604.01870/) (2016)
- [An orthogonalization-free parallelizable framework for all-electron calculations in density functional theory](https://mlsanity.com/paper/2007.14228/) (2020)
- [On the Quality of First-Order Approximation of Functions with H"older Continuous Gradient](https://mlsanity.com/paper/2001.07946/) (2020)

## Built on by (most cited first, 7 of 7)

- [Iso-Riemannian Optimization on Learned Data Manifolds](https://mlsanity.com/paper/2510.21033/) (2025)
- [Efficient Optimization with Orthogonality Constraint: a Randomized Riemannian Submanifold Method](https://mlsanity.com/paper/2505.12378/) (2025)
- [An Embarrassingly Simple Way to Optimize Orthogonal Matrices at Scale](https://mlsanity.com/paper/2602.14656/) (2026)
- [A second-order method landing on the Stiefel manifold via Newtonunicodex2013Schulz iteration](https://mlsanity.com/paper/2605.02838/) (2026)
- [Optimization over the intersection of manifolds](https://mlsanity.com/paper/2605.22736/) (2026)
- [Retraction-Free Optimization over the Stiefel Manifold for the LoRA Fine-Tuning](https://mlsanity.com/paper/2607.25299/) (2026)
- [Online Supervised Dimension Reduction with Random Features: Diagnostics and Computational Trade-offs](https://mlsanity.com/paper/2609.20454/) (2026)

## Abstract (arXiv)

Optimization over the set of matrices X that satisfy X^top B X = Ip, referred to as the generalized Stiefel manifold, appears in many applications involving sampled covariance matrices such as the canonical correlation analysis (CCA), independent component analysis (ICA), and the generalized eigenvalue problem (GEVP). Solving these problems is typically done by iterative methods that require a fully formed B. We propose a cheap stochastic iterative method that solves the optimization problem while having access only to random estimates of B. Our method does not enforce the constraint in every iteration; instead, it produces iterations that converge to critical points on the generalized Stiefel manifold defined in expectation. The method has lower per-iteration cost, requires only matrix multiplications, and has the same convergence rates as its Riemannian optimization counterparts that require the full matrix B. Experiments demonstrate its effectiveness in various machine learning applications involving generalized orthogonality constraints, including CCA, ICA, and the GEVP.