mlsanityData to 7 Oct 2026

Fast and accurate optimization on the orthogonal manifold without retraction

arXiv 2102.07432 ↗First posted 15 Feb 2021stat.ML cs.LG stat.COcited by 35

Not on the map. Why?

Abstract (arXiv)

We consider the problem of minimizing a function over the manifold of orthogonal matrices. The majority of algorithms for this problem compute a direction in the tangent space, and then use a retraction to move in that direction while staying on the manifold. Unfortunately, the numerical computation of retractions on the orthogonal manifold always involves some expensive linear algebra operation, such as matrix inversion, exponential or square-root. These operations quickly become expensive as the dimension of the matrices grows. To bypass this limitation, we propose the landing algorithm which does not use retractions. The algorithm is not constrained to stay on the manifold but its evolution is driven by a potential energy which progressively attracts it towards the manifold. One iteration of the landing algorithm only involves matrix multiplications, which makes it cheap compared to its retraction counterparts. We provide an analysis of the convergence of the algorithm, and demonstrate its promises on large-scale and deep learning problems, where it is faster and less prone to numerical errors than retraction-based methods.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Builds on 18 of its 42 references that are in the map. First posted on arXiv 15 Feb 2021.

Builds on

18 of 42 references.

References · 18

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 526
  6. 2015Unitary Evolution Recurrent Neural Networkscited by 291
  7. 2019Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Groupcited by 133
  8. 2016First-order Methods for Geodesically Convex Optimizationcited by 92
  9. 2017Orthogonal Recurrent Neural Networks with Scaled Cayley Transformcited by 82
  10. 2020Efficient Riemannian Optimization on the Stiefel Manifold via the Cayley Transformcited by 79
  11. 2019Trivializations for Gradient-Based Optimization on Manifoldscited by 78
  12. 2018Can We Gain More from Orthogonality Regularizations in Training Deep CNNs?cited by 62
6 more
  1. 2016Regularizing CNNs with Locally Constrained Decorrelationscited by 41
  2. 2018Averaging Stochastic Gradient Descent on Riemannian Manifoldscited by 39
  3. 2018Parallelizable Algorithms for Optimization Problems with Orthogonality Constraintscited by 16
  4. 2018Towards Riemannian Accelerated Gradient Methodscited by 12
  5. 2017Faster ICA under orthogonal constraintcited by 9
  6. 2019Gradient projection and conditional gradient methods for constrained nonconvex minimizationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 1; 2023: 9; 2024: 7; 2025: 8; 2026: 9.

35 papers.

Sort:

By area:

  1. 2023Diffusion Models for Constrained Domainscited by 60
  2. 2021Pay attention to your loss: understanding misconceptions about 1-Lipschitz neural networkscited by 30
  3. 2025PoLAR: Polar-Decomposed Low-Rank Adapter Representationcited by 19
  4. 2023Infeasible Deterministic, Stochastic, and Variance-Reduction Algorithms for Optimization under Orthogonality Constraintscited by 18
  5. 2023The Edge of Orthogonality: A Simple View of What Makes BYOL Tickcited by 15
  6. 2023Stabilized Neural Differential Equations for Learning Dynamics with Explicit Constraintscited by 15
  7. 2023Orthogonal Directions Constrained Gradient Method: from non-linear equality constraints to Stiefel manifoldcited by 13
  8. 2023Geometry-aware training of factorized layers in tensor Tucker formatcited by 12
  9. 2023Robust low-rank training via approximate orthonormal constraintscited by 12
  10. 2022Momentum Stiefel Optimizer, with Applications to Suitably-Orthogonal Attention, and Optimal Transportcited by 10
  11. 2024Bounding the Expected Robustness of Graph Neural Networks Subject to Node Feature Attackscited by 9
  12. 2024Optimization without Retraction on the Random Generalized Stiefel Manifoldcited by 7
23 more
  1. 2025Landing with the Score: Riemannian Optimization through Denoisingcited by 7
  2. 2024Stiefel Flow Matching for Moment-Constrained Structure Elucidationcited by 5
  3. 2025How to Square Tensor Networks and Circuits Without Squaring Themcited by 5
  4. 2024Riemannian coordinate descent algorithms on matrix manifoldscited by 4
  5. 2025Iso-Riemannian Optimization on Learned Data Manifoldscited by 4
  6. 2025Efficient Optimization with Orthogonality Constraint: a Randomized Riemannian Submanifold Methodcited by 3
  7. 2026An Embarrassingly Simple Way to Optimize Orthogonal Matrices at Scalecited by 3
  8. 2023A Block Coordinate Descent Method for Nonsmooth Composite Optimization under Orthogonality Constraintscited by 2
  9. 2026LipNeXt: Scaling up Lipschitz-based Certified Robustness to Billion-parameter Modelscited by 2
  10. 2026Scalable Variational Bayesian Fine-Tuning of LLMs via Orthogonalized Low-Rank Adapterscited by 2
  11. 2026Low-Rank Adaptation Redux for Large Modelscited by 2
  12. 2024Block-Diagonal Orthogonal Relation and Matrix Entity for Knowledge Graph Embeddingcited by 1
  13. 2024Oja's plasticity rule overcomes several challenges of training neural networks under biological constraintscited by 1
  14. 2025Decentralized Optimization on Compact Submanifolds by Quantized Riemannian Gradient Trackingcited by 1
  15. 2025Feedback Alignment Meets Low-Rank Manifolds: A Structured Recipe for Local Learningcited by 1
  16. 2026Primal-Dual Methods for Nonsmooth Nonconvex Optimization with Orthogonality Constraintscited by 1
  17. 2023Convex optimization over a probability simplexcited by 0
  18. 2024Convergence and Complexity Guarantee for Inexact First-order Riemannian Optimization Algorithmscited by 0
  19. 2025Fast dynamical similarity analysiscited by 0
  20. 2026Efficient Diffusion Models under Nonconvex Equality and Inequality constraints via Landingcited by 0
  21. 2026A second-order method landing on the Stiefel manifold via Newtonunicodex2013Schulz iterationcited by 0
  22. 2026Optimization over the intersection of manifoldscited by 0
  23. 2026Retraction-Free Optimization over the Stiefel Manifold for the LoRA Fine-Tuningcited by 0