mlsanityData to 7 Oct 2026

Orthogonal Directions Constrained Gradient Method: from non-linear equality constraints to Stiefel manifold

arXiv 2303.09261 ↗First posted 16 Mar 2023math.OC stat.MLcited by 13

Not on the map. Why?

Abstract (arXiv)

We consider the problem of minimizing a non-convex function over a smooth manifold M. We propose a novel algorithm, the Orthogonal Directions Constrained Gradient Method (ODCGM) which only requires computing a projection onto a vector space. ODCGM is infeasible but the iterates are constantly pulled towards the manifold, ensuring the convergence of ODCGM towards M. ODCGM is much simpler to implement than the classical methods which require the computation of a retraction. Moreover, we show that ODCGM exhibits the near-optimal oracle complexities O(1/varepsilon2) and O(1/varepsilon4) in the deterministic and stochastic cases, respectively. Furthermore, we establish that, under an appropriate choice of the projection metric, our method recovers the landing algorithm of Ablin and Peyr'e (2022), a recently introduced algorithm for optimization over the Stiefel manifold. As a result, we significantly extend the analysis of Ablin and Peyr'e (2022), establishing near-optimal rates both in deterministic and stochastic frameworks. Finally, we perform numerical experiments which shows the efficiency of ODCGM in a high-dimensional setting.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 19 of its 40 references that are in the map. First posted on arXiv 16 Mar 2023.

Builds on

19 of 40 references.

References · 19

  1. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 716
  2. 2019Lower Bounds for Non-Convex Stochastic Optimizationcited by 303
  3. 2015Unitary Evolution Recurrent Neural Networkscited by 291
  4. 2018Stochastic model-based minimization of weakly convex functionscited by 157
  5. 2017Lower Bounds for Finding Stationary Points Icited by 125
  6. 2016First-order Methods for Geodesically Convex Optimizationcited by 92
  7. 2013Accelerated Gradient Methods for Nonconvex Nonlinear and Stochastic Programmingcited by 74
  8. 2019Orthogonal Deep Neural Networkscited by 72
  9. 2018Can We Gain More from Orthogonality Regularizations in Training Deep CNNs?cited by 62
  10. 2016Global rates of convergence for nonconvex optimization on manifoldscited by 61
  11. 2020Deep Isometric Learning for Visual Recognitioncited by 38
  12. 2021Fast and accurate optimization on the orthogonal manifold without retractioncited by 35
7 more
  1. 2020Rate-improved Inexact Augmented Lagrangian Method for Constrained Nonconvex Optimizationcited by 21
  2. 2019Complexity of Proximal augmented Lagrangian for nonconvex optimization with nonlinear equality constraintscited by 17
  3. 2017Faster independent component analysis by preconditioning with Hessian approximationscited by 16
  4. 2021On Constraints in First-Order Optimization: A View from Non-Smooth Dynamical Systemscited by 13
  5. 2022Optimization flows landing on the Stiefel manifoldcited by 10
  6. 2017Optimality condition and complexity analysis for linearly-constrained optimization without differentiability on the boundarycited by 8
  7. 2022AskewSGD : An Annealed interval-constrained Optimisation method to train Quantized Neural Networkscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 2; 2025: 5; 2026: 5.

13 papers.

Sort:

By area:

  1. 2025PoLAR: Polar-Decomposed Low-Rank Adapter Representationcited by 19
  2. 2023Infeasible Deterministic, Stochastic, and Variance-Reduction Algorithms for Optimization under Orthogonality Constraintscited by 18
  3. 2024Optimization without Retraction on the Random Generalized Stiefel Manifoldcited by 7
  4. 2025Landing with the Score: Riemannian Optimization through Denoisingcited by 7
  5. 2024Primal Methods for Variational Inequality Problems with Functional Constraintscited by 3
  6. 2026Scalable Variational Bayesian Fine-Tuning of LLMs via Orthogonalized Low-Rank Adapterscited by 2
  7. 2026Low-Rank Adaptation Redux for Large Modelscited by 2
  8. 2025Safe Gradient Flow for Bilevel Optimizationcited by 1
  9. 2025Sequential QCQP for Bilevel Optimization with Line Searchcited by 0
  10. 2025A Systems-Theoretic View on the Convergence of Algorithms under Disturbancescited by 0
  11. 2026Efficient Diffusion Models under Nonconvex Equality and Inequality constraints via Landingcited by 0
  12. 2026A second-order method landing on the Stiefel manifold via Newtonunicodex2013Schulz iterationcited by 0
1 more
  1. 2026Optimization over the intersection of manifoldscited by 0