mlsanityData to 7 Oct 2026

Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNs

arXiv 1612.05231 ↗First posted 15 Dec 2016cs.LG cs.NE stat.MLcited by 62

Not on the map. Why?

Abstract (arXiv)

Using unitary (instead of general) matrices in artificial neural networks (ANNs) is a promising way to solve the gradient explosion/vanishing problem, as well as to enable ANNs to learn long-term correlations in the data. This approach appears particularly promising for Recurrent Neural Networks (RNNs). In this work, we present a new architecture for implementing an Efficient Unitary Neural Network (EUNNs); its main advantages can be summarized as follows. Firstly, the representation capacity of the unitary space in an EUNN is fully tunable, ranging from a subspace of SU(N) to the entire unitary space. Secondly, the computational complexity for training an EUNN is merely O(1) per parameter. Finally, we test the performance of EUNNs on the standard copying task, the pixel-permuted MNIST digit recognition benchmark as well as the Speech Prediction Test (TIMIT). We find that our architecture significantly outperforms both other state-of-the-art unitary RNNs and the LSTM architecture, in terms of the final performance and/or the wall-clock training speed. EUNNs are thus promising alternatives to RNNs and LSTMs for a wide variety of applications.

Key facts. Cited by 62 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Dec 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 17; 2022: 14; 2023: 3; 2024: 10; 2025: 8; 2026: 10.

62 papers.

Sort:

By area:

  1. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
  2. 2022Omnigrok: Grokking Beyond Algorithmic Datacited by 142
  3. 2024Barren Plateaus in Variational Quantum Computingcited by 130
  4. 2023Parameter-Efficient Orthogonal Finetuning via Butterfly Factorizationcited by 112
  5. 2021Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Modelscited by 75
  6. 2025DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Productscited by 69
  7. 2022Quantum Vision Transformerscited by 48
  8. 2021Heavy Ball Neural Ordinary Differential Equationscited by 42
  9. 2021Learning with Hyperspherical Uniformitycited by 35
  10. 2024Unlocking the Power of LSTM for Long Term Time Series Forecastingcited by 29
  11. 2022projUNN: efficient method for training deep networks with unitary matricescited by 27
  12. 2021Improving Molecular Graph Neural Network Explainability with Orthonormalization and Induced Sparsitycited by 20
38 more
  1. 2021MC-LSTM: Mass-Conserving LSTMcited by 17
  2. 2021Recurrent Neural Network from Adder's Perspective: Carry-lookahead RNNcited by 16
  3. 2021Learning quantum data with the quantum Earth Mover's distancecited by 13
  4. 2021Implicit Bias of Linear RNNscited by 11
  5. 2022DOSnet as a Non-Black-Box PDE Solver: When Deep Learning Meets Operator Splittingcited by 11
  6. 2022RF-Next: Efficient Receptive Field Search for Convolutional Neural Networkscited by 10
  7. 2025Quantum-PEFT: Ultra parameter-efficient fine-tuningcited by 9
  8. 2024Unitary convolutions for learning on graphs and groupscited by 8
  9. 2021Slower is Better: Revisiting the Forgetting Mechanism in LSTM for Slower Information Decaycited by 7
  10. 2021Scaling-up Diverse Orthogonal Convolutional Networks with a Paraunitary Frameworkcited by 7
  11. 2026Deep Delta Learningcited by 7
  12. 2024Neural Gaussian Scale-Space Fieldscited by 6
  13. 2024RotRNN: Modelling Long Sequences with Rotationscited by 6
  14. 2021Coordinate descent on the orthogonal group for recurrent neural network trainingcited by 5
  15. 2022Feedback Gradient Descent: Efficient and Stable Optimization with Orthogonality for DNNscited by 5
  16. 2022Random orthogonal additive filters: a solution to the vanishing/exploding gradient of deep neural networkscited by 5
  17. 2024Positional Encoding Helps Recurrent Neural Networks Handle a Large Vocabularycited by 5
  18. 2024Graph Unitary Message Passingcited by 5
  19. 2024Approximately-symmetric neural networks for quantum spin liquidscited by 5
  20. 2022Entangled Residual Mappingscited by 4
  21. 2022Orthogonal Gated Recurrent Unit with Neumann-Cayley Transformationcited by 4
  22. 2025Emergence of the Primacy Effect in Structured State-Space Modelscited by 4
  23. 2021A Differential Geometry Perspective on Orthogonal Recurrent Modelscited by 3
  24. 2022Lyapunov-Guided Representation of Recurrent Neural Network Performancecited by 3
  25. 2022Learning to Optimize Quasi-Newton Methodscited by 3
  26. 2024IGNN-Solver: A Graph Neural Solver for Implicit Graph Neural Networkscited by 3
  27. 2021Deep Unitary Convolutional Neural Networkscited by 2
  28. 2021Efficient Identification of Butterfly Sparse Matrix Factorizationscited by 2
  29. 2022Assessing the Unitary RNN as an End-to-End Compositional Model of Syntaxcited by 2
  30. 2025Spectral Alignment as Predictor of Loss Explosion in Neural Network Trainingcited by 2
  31. 2021RotLSTM: Rotating Memories in Recurrent Neural Networkscited by 1
  32. 2021Parallelized Computation and Backpropagation Under Angle-Parametrized Orthogonal Matricescited by 1
  33. 2022Arithmetic Circuits, Structured Matrices and (not so) Deep Learningcited by 1
  34. 2024Quantized Approximately Orthogonal Recurrent Neural Networkscited by 1
  35. 2026The Quantum Shortcut: Complex Phase-State Dynamics Reduce the Optimization Steps of Sequence Modelscited by 1
  36. 2021Acceleration Method for Learning Fine-Layered Optical Neural Networkscited by 0
  37. 2022Estimating the randomness of quantum circuit ensembles up to 50 qubitscited by 0
  38. 2023Adaptive-saturated RNN: Remember more with less instabilitycited by 0

50 most cited of 62 All 62 (JSON)