# Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNs

arXiv: https://arxiv.org/abs/1612.05231 · HTML: https://mlsanity.com/paper/1612.05231/

## Key facts

- Cited by 65 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Dec 2016.

## Built on by (most cited first, 30 of 65)

- [Resurrecting Recurrent Neural Networks for Long Sequences](https://mlsanity.com/paper/2303.06349/) (2023)
- [Omnigrok: Grokking Beyond Algorithmic Data](https://mlsanity.com/paper/2210.01117/) (2022)
- [Barren Plateaus in Variational Quantum Computing](https://mlsanity.com/paper/2405.00781/) (2024)
- [Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization](https://mlsanity.com/paper/2311.06243/) (2023)
- [Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models](https://mlsanity.com/paper/2112.00029/) (2021)
- [DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products](https://mlsanity.com/paper/2502.10297/) (2025)
- [Quantum Vision Transformers](https://mlsanity.com/paper/2209.08167/) (2022)
- [Heavy Ball Neural Ordinary Differential Equations](https://mlsanity.com/paper/2110.04840/) (2021)
- [Learning with Hyperspherical Uniformity](https://mlsanity.com/paper/2103.01649/) (2021)
- [Unlocking the Power of LSTM for Long Term Time Series Forecasting](https://mlsanity.com/paper/2408.10006/) (2024)
- [projUNN: efficient method for training deep networks with unitary matrices](https://mlsanity.com/paper/2203.05483/) (2022)
- [Improving Molecular Graph Neural Network Explainability with Orthonormalization and Induced Sparsity](https://mlsanity.com/paper/2105.04854/) (2021)
- [MC-LSTM: Mass-Conserving LSTM](https://mlsanity.com/paper/2101.05186/) (2021)
- [Recurrent Neural Network from Adder's Perspective: Carry-lookahead RNN](https://mlsanity.com/paper/2106.12901/) (2021)
- [Learning quantum data with the quantum Earth Mover's distance](https://mlsanity.com/paper/2101.03037/) (2021)
- [Implicit Bias of Linear RNNs](https://mlsanity.com/paper/2101.07833/) (2021)
- [DOSnet as a Non-Black-Box PDE Solver: When Deep Learning Meets Operator Splitting](https://mlsanity.com/paper/2212.05571/) (2022)
- [Quantum-PEFT: Ultra parameter-efficient fine-tuning](https://mlsanity.com/paper/2503.05431/) (2025)
- [RF-Next: Efficient Receptive Field Search for Convolutional Neural Networks](https://mlsanity.com/paper/2206.06637/) (2022)
- [Deep Delta Learning](https://mlsanity.com/paper/2601.00417/) (2026)
- [Unitary convolutions for learning on graphs and groups](https://mlsanity.com/paper/2410.05499/) (2024)
- [Slower is Better: Revisiting the Forgetting Mechanism in LSTM for Slower Information Decay](https://mlsanity.com/paper/2105.05944/) (2021)
- [Scaling-up Diverse Orthogonal Convolutional Networks with a Paraunitary Framework](https://mlsanity.com/paper/2106.09121/) (2021)
- [Neural Gaussian Scale-Space Fields](https://mlsanity.com/paper/2405.20980/) (2024)
- [RotRNN: Modelling Long Sequences with Rotations](https://mlsanity.com/paper/2407.07239/) (2024)
- [Coordinate descent on the orthogonal group for recurrent neural network training](https://mlsanity.com/paper/2108.00051/) (2021)
- [Feedback Gradient Descent: Efficient and Stable Optimization with Orthogonality for DNNs](https://mlsanity.com/paper/2205.08385/) (2022)
- [Random orthogonal additive filters: a solution to the vanishing/exploding gradient of deep neural networks](https://mlsanity.com/paper/2210.01245/) (2022)
- [Positional Encoding Helps Recurrent Neural Networks Handle a Large Vocabulary](https://mlsanity.com/paper/2402.00236/) (2024)
- [Graph Unitary Message Passing](https://mlsanity.com/paper/2403.11199/) (2024)

## Abstract (arXiv)

Using unitary (instead of general) matrices in artificial neural networks (ANNs) is a promising way to solve the gradient explosion/vanishing problem, as well as to enable ANNs to learn long-term correlations in the data. This approach appears particularly promising for Recurrent Neural Networks (RNNs). In this work, we present a new architecture for implementing an Efficient Unitary Neural Network (EUNNs); its main advantages can be summarized as follows. Firstly, the representation capacity of the unitary space in an EUNN is fully tunable, ranging from a subspace of SU(N) to the entire unitary space. Secondly, the computational complexity for training an EUNN is merely O(1) per parameter. Finally, we test the performance of EUNNs on the standard copying task, the pixel-permuted MNIST digit recognition benchmark as well as the Speech Prediction Test (TIMIT). We find that our architecture significantly outperforms both other state-of-the-art unitary RNNs and the LSTM architecture, in terms of the final performance and/or the wall-clock training speed. EUNNs are thus promising alternatives to RNNs and LSTMs for a wide variety of applications.