# Orthogonal Directions Constrained Gradient Method: from non-linear equality constraints to Stiefel manifold

arXiv: https://arxiv.org/abs/2303.09261 · HTML: https://mlsanity.com/paper/2303.09261/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 40 references that are in the map.
- First posted on arXiv 16 Mar 2023.

## Builds on (most cited first, 19 of 19)

- [Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programming](https://mlsanity.com/paper/1309.5549/) (2013)
- [Lower Bounds for Non-Convex Stochastic Optimization](https://mlsanity.com/paper/1912.02365/) (2019)
- [Unitary Evolution Recurrent Neural Networks](https://mlsanity.com/paper/1511.06464/) (2015)
- [Stochastic model-based minimization of weakly convex functions](https://mlsanity.com/paper/1803.06523/) (2018)
- [Lower Bounds for Finding Stationary Points I](https://mlsanity.com/paper/1710.11606/) (2017)
- [First-order Methods for Geodesically Convex Optimization](https://mlsanity.com/paper/1602.06053/) (2016)
- [Accelerated Gradient Methods for Nonconvex Nonlinear and Stochastic Programming](https://mlsanity.com/paper/1310.3787/) (2013)
- [Orthogonal Deep Neural Networks](https://mlsanity.com/paper/1905.05929/) (2019)
- [Global rates of convergence for nonconvex optimization on manifolds](https://mlsanity.com/paper/1605.08101/) (2016)
- [Can We Gain More from Orthogonality Regularizations in Training Deep CNNs?](https://mlsanity.com/paper/1810.09102/) (2018)
- [Deep Isometric Learning for Visual Recognition](https://mlsanity.com/paper/2006.16992/) (2020)
- [Fast and accurate optimization on the orthogonal manifold without retraction](https://mlsanity.com/paper/2102.07432/) (2021)
- [Rate-improved Inexact Augmented Lagrangian Method for Constrained Nonconvex Optimization](https://mlsanity.com/paper/2007.01284/) (2020)
- [Complexity of Proximal augmented Lagrangian for nonconvex optimization with nonlinear equality constraints](https://mlsanity.com/paper/1908.00131/) (2019)
- [Faster independent component analysis by preconditioning with Hessian approximations](https://mlsanity.com/paper/1706.08171/) (2017)
- [On Constraints in First-Order Optimization: A View from Non-Smooth Dynamical Systems](https://mlsanity.com/paper/2107.08225/) (2021)
- [Optimization flows landing on the Stiefel manifold](https://mlsanity.com/paper/2202.09058/) (2022)
- [Optimality condition and complexity analysis for linearly-constrained optimization without differentiability on the boundary](https://mlsanity.com/paper/1702.04300/) (2017)
- [AskewSGD : An Annealed interval-constrained Optimisation method to train Quantized Neural Networks](https://mlsanity.com/paper/2211.03741/) (2022)

## Built on by (most cited first, 14 of 14)

- [PoLAR: Polar-Decomposed Low-Rank Adapter Representation](https://mlsanity.com/paper/2506.03133/) (2025)
- [Infeasible Deterministic, Stochastic, and Variance-Reduction Algorithms for Optimization under Orthogonality Constraints](https://mlsanity.com/paper/2303.16510/) (2023)
- [Landing with the Score: Riemannian Optimization through Denoising](https://mlsanity.com/paper/2509.23357/) (2025)
- [Optimization without Retraction on the Random Generalized Stiefel Manifold](https://mlsanity.com/paper/2405.01702/) (2024)
- [Primal Methods for Variational Inequality Problems with Functional Constraints](https://mlsanity.com/paper/2403.12859/) (2024)
- [Low-Rank Adaptation Redux for Large Models](https://mlsanity.com/paper/2604.21905/) (2026)
- [Scalable Variational Bayesian Fine-Tuning of LLMs via Orthogonalized Low-Rank Adapters](https://mlsanity.com/paper/2604.03388/) (2026)
- [Safe Gradient Flow for Bilevel Optimization](https://mlsanity.com/paper/2501.16520/) (2025)
- [Sequential QCQP for Bilevel Optimization with Line Search](https://mlsanity.com/paper/2505.14647/) (2025)
- [A Systems-Theoretic View on the Convergence of Algorithms under Disturbances](https://mlsanity.com/paper/2512.17598/) (2025)
- [Efficient Diffusion Models under Nonconvex Equality and Inequality constraints via Landing](https://mlsanity.com/paper/2604.17838/) (2026)
- [A second-order method landing on the Stiefel manifold via Newtonunicodex2013Schulz iteration](https://mlsanity.com/paper/2605.02838/) (2026)
- [Optimization over the intersection of manifolds](https://mlsanity.com/paper/2605.22736/) (2026)
- [Learning to Select Features with Manifold Deep Unfolding PCA](https://mlsanity.com/paper/2610.19436/) (2026)

## Abstract (arXiv)

We consider the problem of minimizing a non-convex function over a smooth manifold M. We propose a novel algorithm, the Orthogonal Directions Constrained Gradient Method (ODCGM) which only requires computing a projection onto a vector space. ODCGM is infeasible but the iterates are constantly pulled towards the manifold, ensuring the convergence of ODCGM towards M. ODCGM is much simpler to implement than the classical methods which require the computation of a retraction. Moreover, we show that ODCGM exhibits the near-optimal oracle complexities O(1/varepsilon2) and O(1/varepsilon4) in the deterministic and stochastic cases, respectively. Furthermore, we establish that, under an appropriate choice of the projection metric, our method recovers the landing algorithm of Ablin and Peyr'e (2022), a recently introduced algorithm for optimization over the Stiefel manifold. As a result, we significantly extend the analysis of Ablin and Peyr'e (2022), establishing near-optimal rates both in deterministic and stochastic frameworks. Finally, we perform numerical experiments which shows the efficiency of ODCGM in a high-dimensional setting.