# Parallelized Computation and Backpropagation Under Angle-Parametrized Orthogonal Matrices

arXiv: https://arxiv.org/abs/2106.00003 · HTML: https://mlsanity.com/paper/2106.00003/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 17 references that are in the map.
- First posted on arXiv 30 May 2021.

## Builds on (most cited first, 11 of 11)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Normalizing Flows for Probabilistic Modeling and Inference](https://mlsanity.com/paper/1912.02762/) (2019)
- [Unitary Evolution Recurrent Neural Networks](https://mlsanity.com/paper/1511.06464/) (2015)
- [Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group](https://mlsanity.com/paper/1901.08428/) (2019)
- [Full-Capacity Unitary Recurrent Neural Networks](https://mlsanity.com/paper/1611.00035/) (2016)
- [Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networks](https://mlsanity.com/paper/1709.06079/) (2017)
- [Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNs](https://mlsanity.com/paper/1612.05231/) (2016)
- [Generalized BackPropagation, 'Etude De Cas: Orthogonality](https://mlsanity.com/paper/1611.05927/) (2016)
- [Optimization on Submanifolds of Convolution Kernels in CNNs](https://mlsanity.com/paper/1610.07008/) (2016)
- [Fast Approximation of Rotations and Hessians matrices](https://mlsanity.com/paper/1404.7195/) (2014)

## Built on by (most cited first, 1 of 1)

- [Training-efficient density quantum machine learning](https://mlsanity.com/paper/2405.20237/) (2024)

## Abstract (arXiv)

We present a methodology for parallel acceleration of learning in the presence of matrix orthogonality and unitarity constraints of interest in several branches of machine learning. We show how an apparently sequential elementary rotation parametrization can be restructured into blocks of commutative operations using a well-known tool for coloring the edges of complete graphs, in turn widely applied to schedule round-robin (all-against-all) sports tournaments. The resulting decomposition admits an algorithm to compute a fully-parametrized orthogonal matrix from its rotation parameters in O(n) sequential steps and one to compute the gradient of a training loss with respect to its parameters in O(nlog n) steps. We discuss parametric restrictions of interest to generative modeling and present promising performance results with a prototype GPU implementation.