# projUNN: efficient method for training deep networks with unitary matrices

arXiv: https://arxiv.org/abs/2203.05483 · HTML: https://mlsanity.com/paper/2203.05483/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 53 of its 110 references that are in the map.
- First posted on arXiv 10 Mar 2022.

## Builds on (most cited first, 30 of 53)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Variational Quantum Algorithms](https://mlsanity.com/paper/2012.09265/) (2020)
- [Barren plateaus in quantum neural network training landscapes](https://mlsanity.com/paper/1803.11173/) (2018)
- [HiPPO: Recurrent Memory with Optimal Polynomial Projections](https://mlsanity.com/paper/2008.07669/) (2020)
- [Implicit Regularization in Deep Matrix Factorization](https://mlsanity.com/paper/1905.13655/) (2019)
- [Implicit Regularization in Matrix Factorization](https://mlsanity.com/paper/1705.09280/) (2017)
- [Stronger generalization bounds for deep nets via a compression approach](https://mlsanity.com/paper/1802.05296/) (2018)
- [Unitary Evolution Recurrent Neural Networks](https://mlsanity.com/paper/1511.06464/) (2015)
- [Speeding up Convolutional Neural Networks with Low Rank Expansions](https://mlsanity.com/paper/1405.3866/) (2014)
- [PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization](https://mlsanity.com/paper/1905.13727/) (2019)
- [Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning](https://mlsanity.com/paper/1810.01075/) (2018)
- [Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks](https://mlsanity.com/paper/1806.05393/) (2018)
- [Efficient Learning for Deep Quantum Neural Networks](https://mlsanity.com/paper/1902.10445/) (2019)
- [Understanding Batch Normalization](https://mlsanity.com/paper/1806.02375/) (2018)
- [Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group](https://mlsanity.com/paper/1901.08428/) (2019)
- [Orthogonal Convolutional Neural Networks](https://mlsanity.com/paper/1911.12207/) (2019)
- [AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks](https://mlsanity.com/paper/1902.09689/) (2019)
- [Orthogonalizing Convolutional Layers with the Cayley Transform](https://mlsanity.com/paper/2104.07167/) (2021)
- [Convolutional neural networks with low-rank regularization](https://mlsanity.com/paper/1511.06067/) (2015)
- [The Singular Values of Convolutional Layers](https://mlsanity.com/paper/1805.10408/) (2018)
- [Full-Capacity Unitary Recurrent Neural Networks](https://mlsanity.com/paper/1611.00035/) (2016)
- [Fast Training of Convolutional Networks through FFTs](https://mlsanity.com/paper/1312.5851/) (2013)
- [Lipschitz Recurrent Neural Networks](https://mlsanity.com/paper/2006.12070/) (2020)
- [Preventing Gradient Attenuation in Lipschitz Constrained Convolutional Networks](https://mlsanity.com/paper/1911.00937/) (2019)
- [Orthogonal Recurrent Neural Networks with Scaled Cayley Transform](https://mlsanity.com/paper/1707.09520/) (2017)
- [Efficient Riemannian Optimization on the Stiefel Manifold via the Cayley Transform](https://mlsanity.com/paper/2002.01113/) (2020)
- [Continuous-variable quantum neural networks](https://mlsanity.com/paper/1806.06871/) (2018)

## Built on by (most cited first, 27 of 27)

- [Quantum Vision Transformers](https://mlsanity.com/paper/2209.08167/) (2022)
- [Backward Lens: Projecting Language Model Gradients into the Vocabulary Space](https://mlsanity.com/paper/2402.12865/) (2024)
- [Self-Supervised Learning with Lie Symmetries for Partial Differential Equations](https://mlsanity.com/paper/2307.05432/) (2023)
- [Stabilized Neural Differential Equations for Learning Dynamics with Explicit Constraints](https://mlsanity.com/paper/2306.09739/) (2023)
- [Efficient Storage of Fine-Tuned Models via Low-Rank Approximation of Weight Residuals](https://mlsanity.com/paper/2305.18425/) (2023)
- [DOSnet as a Non-Black-Box PDE Solver: When Deep Learning Meets Operator Splitting](https://mlsanity.com/paper/2212.05571/) (2022)
- [Training-efficient density quantum machine learning](https://mlsanity.com/paper/2405.20237/) (2024)
- [Improved techniques for deterministic l2 robustness](https://mlsanity.com/paper/2211.08453/) (2022)
- [Path Development Network with Finite-dimensional Lie Group Representation](https://mlsanity.com/paper/2204.00740/) (2022)
- [Edit at your own risk: evaluating the robustness of edited models to distribution shifts](https://mlsanity.com/paper/2303.00046/) (2023)
- [Unitary convolutions for learning on graphs and groups](https://mlsanity.com/paper/2410.05499/) (2024)
- [An Adaptive Orthogonal Convolution Scheme for Efficient and Flexible CNN Architectures](https://mlsanity.com/paper/2501.07930/) (2025)
- [Generating Universal Adversarial Perturbations for Quantum Classifiers](https://mlsanity.com/paper/2402.08648/) (2024)
- [Graph Unitary Message Passing](https://mlsanity.com/paper/2403.11199/) (2024)
- [An Embarrassingly Simple Way to Optimize Orthogonal Matrices at Scale](https://mlsanity.com/paper/2602.14656/) (2026)
- [Assessing the Unitary RNN as an End-to-End Compositional Model of Syntax](https://mlsanity.com/paper/2208.05719/) (2022)
- [Optimal Quantum Circuit Design via Unitary Neural Networks](https://mlsanity.com/paper/2408.13211/) (2024)
- [Spectral Alignment as Predictor of Loss Explosion in Neural Network Training](https://mlsanity.com/paper/2510.04202/) (2025)
- [Quantized Approximately Orthogonal Recurrent Neural Networks](https://mlsanity.com/paper/2402.04012/) (2024)
- [Hybrid Quantum-Classical Recurrent Neural Networks](https://mlsanity.com/paper/2510.25557/) (2025)
- [U(1) Symmetry-breaking Observed in Generic CNN Bottleneck Layers](https://mlsanity.com/paper/2206.02220/) (2022)
- [Testing predictions of representation cost theory with CNNs](https://mlsanity.com/paper/2210.01257/) (2022)
- [Coloring Deep CNN Layers with Activation Hue Loss](https://mlsanity.com/paper/2310.03911/) (2023)
- [Neural Controlled Differential Equations with Quantum Hidden Evolutions](https://mlsanity.com/paper/2404.19673/) (2024)
- [Input-driven circuit reconfiguration in critical recurrent neural networks](https://mlsanity.com/paper/2405.15036/) (2024)
- [HadamRNN: Binary and Sparse Ternary Orthogonal RNNs](https://mlsanity.com/paper/2502.00047/) (2025)
- [Orthogonium : A Unified, Efficient Library of Orthogonal and 1-Lipschitz Building Blocks](https://mlsanity.com/paper/2601.13776/) (2026)

## Abstract (arXiv)

In learning with recurrent or very deep feed-forward networks, employing unitary matrices in each layer can be very effective at maintaining long-range stability. However, restricting network parameters to be unitary typically comes at the cost of expensive parameterizations or increased training runtime. We propose instead an efficient method based on rank-k updates -- or their rank-k approximation -- that maintains performance at a nearly optimal training runtime. We introduce two variants of this method, named Direct (projUNN-D) and Tangent (projUNN-T) projected Unitary Neural Networks, that can parameterize full N-dimensional unitary or orthogonal matrices with a training runtime scaling as O(kN2). Our method either projects low-rank gradients onto the closest unitary matrix (projUNN-T) or transports unitary matrices in the direction of the low-rank gradient (projUNN-D). Even in the fastest setting (k=1), projUNN is able to train a model's unitary parameters to reach comparable performances against baseline implementations. In recurrent neural network settings, projUNN closely matches or exceeds benchmarked results from prior unitary neural networks. Finally, we preliminarily explore projUNN in training orthogonal convolutional neural networks, which are currently unable to outperform state of the art models but can potentially enhance stability and robustness at large depth.