An Embarrassingly Simple Way to Optimize Orthogonal Matrices at Scale
Not on the map. Why?
Abstract (arXiv)
Orthogonality constraints are ubiquitous in robust and probabilistic machine learning. Unfortunately, current optimizers are computationally expensive and do not scale to problems with hundreds or thousands of constraints. One notable exception is the Landing algorithm (Ablin et al., 2024) which, however comes at the expense of temporarily relaxing orthogonality. In this work, we revisit and improve on the ideas behind Landing, enabling the inclusion of modern adaptive optimizers while ensuring that orthogonal constraints are effectively met. Remarkably, these improvements come at little to no cost, and reduce the number of required hyperparemeters. Our algorithm POGO is fast and GPU-friendly, consisting of only 5 matrix products, and in practice maintains orthogonality at all times. On several challenging benchmarks, POGO greatly outperforms recent optimizers and shows it can optimize problems with thousands of orthogonal matrices in minutes while alternatives would take hours. As such, POGO sets a milestone to finally exploit orthogonality constraints in ML at scale. A PyTorch implementation of POGO is publicly available at https://github.com/adrianjav/pogo.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 23 of its 53 references that are in the map. First posted on arXiv 16 Feb 2026.
Builds on
23 of 53 references.
References · 23
- 2015Unitary Evolution Recurrent Neural Networkscited by 291
- 2019Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Groupcited by 133
- 2019Orthogonal Convolutional Neural Networkscited by 122
- 2019Trivializations for Gradient-Based Optimization on Manifoldscited by 78
- 2019Orthogonal Deep Neural Networkscited by 72
- 2023Subtractive Mixture Models via Squaring: Representation and Learningcited by 39
- 2021Fast and accurate optimization on the orthogonal manifold without retractioncited by 35
- 2022projUNN: efficient method for training deep networks with unitary matricescited by 27
- 2024Sum of Squares Circuitscited by 21
- 2023Infeasible Deterministic, Stochastic, and Variance-Reduction Algorithms for Optimization under Orthogonality Constraintscited by 18
- 2022VectorAdam for Rotation Equivariant Geometry Optimizationcited by 17
- 2018Parallelizable Algorithms for Optimization Problems with Orthogonality Constraintscited by 16
11 more
- 2016Optimization on Submanifolds of Convolution Kernels in CNNscited by 15
- 202494% on CIFAR-10 in 3.29 Seconds on a Single GPUcited by 15
- 2022O-ViT: Orthogonal Vision Transformercited by 9
- 2021A Penalty-free Infeasible Approach for a Class of Nonsmooth Optimization Problems over the Stiefel Manifoldcited by 8
- 2024Optimization without Retraction on the Random Generalized Stiefel Manifoldcited by 7
- 2020Manifold Proximal Point Algorithms for Dual Principal Component Pursuit and Orthogonal Dictionary Learningcited by 6
- 2025How to Square Tensor Networks and Circuits Without Squaring Themcited by 5
- 2020An orthogonalization-free parallelizable framework for all-electron calculations in density functional theorycited by 4
- 2024Riemannian coordinate descent algorithms on matrix manifoldscited by 4
- 2025Efficient Optimization with Orthogonality Constraint: a Randomized Riemannian Submanifold Methodcited by 3
- 2025Distributed Retraction-Free and Communication-Efficient Optimization on the Stiefel Manifoldcited by 3
Built on by
Citing papers per year: 2026: 3.
3 papers.