mlsanityData to 14 Oct 2026

Unitary Evolution Recurrent Neural Networks

arXiv 1511.06464 ↗First posted 20 Nov 2015cs.LG cs.NE stat.MLcited by 294

Not on the map. Why?

Abstract (arXiv)

Recurrent neural networks (RNNs) are notoriously difficult to train. When the eigenvalues of the hidden to hidden weight matrix deviate from absolute value 1, optimization becomes difficult due to the well studied issue of vanishing and exploding gradients, especially when trying to learn long-term dependencies. To circumvent this problem, we propose a new architecture that learns a unitary weight matrix, with eigenvalues of absolute value exactly 1. The challenge we address is that of parametrizing unitary matrices in a way that does not require expensive computations (such as eigendecomposition) after each weight update. We construct an expressive unitary weight matrix by composing several structured matrices that act as building blocks with parameters to be learned. Optimization with this parameterization becomes feasible only when considering hidden states in the complex domain. We demonstrate the potential of this architecture by achieving state of the art results in several hard tasks involving very long-term dependencies.

Key facts. Cited by 294 papers in the map (data through 14 Oct 2026). First posted on arXiv 20 Nov 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 52; 2022: 36; 2023: 47; 2024: 53; 2025: 37; 2026: 69.

294 papers.

Sort:

By area:

  1. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  2. 2022Simplified State Space Layers for Sequence Modelingcited by 788
  3. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 608
  4. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 408
  5. 2021A Universal Law of Robustness via Isoperimetrycited by 151
  6. 2024Barren Plateaus in Variational Quantum Computingcited by 133
  7. 2021CKConv: Continuous Kernel Convolution For Sequential Datacited by 120
  8. 2024Transformers are Multi-State RNNscited by 120
  9. 2021Accurate and efficient time-domain classification with adaptive spiking recurrent neural networkscited by 115
  10. 2021Orthogonalizing Convolutional Layers with the Cayley Transformcited by 105
  11. 2023Hierarchically Gated Recurrent Neural Network for Sequence Modelingcited by 102
  12. 2023A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need?cited by 84
38 more
  1. 2021On the difficulty of learning chaotic dynamics with RNNscited by 67
  2. 2024Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challengescited by 64
  3. 2024fKAN: Fractional Kolmogorov-Arnold Networks with trainable Jacobi basis functionscited by 64
  4. 2021A Survey of Complex-Valued Neural Networkscited by 61
  5. 2023Generalized Teacher Forcing for Learning Chaotic Dynamicscited by 58
  6. 2021UnICORNN: A recurrent model for learning very long time dependenciescited by 56
  7. 2024Utilizing Image Transforms and Diffusion Models for Generative Modeling of Short and Long Time Seriescited by 43
  8. 2025On Vanishing Gradients, Over-Smoothing, and Over-Squashing in GNNs: Bridging Recurrent and Graph Learningcited by 43
  9. 2021Heavy Ball Neural Ordinary Differential Equationscited by 42
  10. 2021Long Expressive Memory for Sequence Modelingcited by 41
  11. 2024OLoRA: Orthonormal Low-Rank Adaptation of Large Language Modelscited by 39
  12. 2024Oscillatory State-Space Modelscited by 39
  13. 2024Were RNNs All We Needed?cited by 38
  14. 2021Fast and accurate optimization on the orthogonal manifold without retractioncited by 35
  15. 2021Learning with Hyperspherical Uniformitycited by 35
  16. 2022The Principle of Diversity: Training Stronger Vision Transformers Calls for Reducing All Levels of Redundancycited by 35
  17. 2021Decentralized Riemannian Gradient Descent on the Stiefel Manifoldcited by 33
  18. 2021Pay attention to your loss: understanding misconceptions about 1-Lipschitz neural networkscited by 30
  19. 2022How important are activation functions in regression and classification? A survey, performance comparison, and future directionscited by 29
  20. 2022Modeling Multivariate Biosignals With Graph Neural Networks and Structured State Space Modelscited by 29
  21. 2023Self-Supervised Learning with Lie Symmetries for Partial Differential Equationscited by 29
  22. 2022Machine Learning for Microcontroller-Class Hardware: A Reviewcited by 28
  23. 2022projUNN: efficient method for training deep networks with unitary matricescited by 27
  24. 2024VkD: Improving Knowledge Distillation using Orthogonal Projectionscited by 26
  25. 2024Recurrent neural networks: vanishing and exploding gradients are not the end of the storycited by 25
  26. 2023T-FFTRadNet: Object Detection with Swin Vision Transformers from Raw ADC Radar Signalscited by 24
  27. 2023A Survey of Geometric Optimization for Deep Learning: From Euclidean Space to Riemannian Manifoldcited by 21
  28. 2023Spectral State Space Modelscited by 21
  29. 2024Sum of Squares Circuitscited by 21
  30. 2021Improving Molecular Graph Neural Network Explainability with Orthonormalization and Induced Sparsitycited by 20
  31. 2021Learning Connectivity with Graph Convolutional Networks for Skeleton-based Action Recognitioncited by 20
  32. 2025Training Transformers with Enforced Lipschitz Constantscited by 20
  33. 2021Hamiltonian Deep Neural Networks Guaranteeing Non-vanishing Gradients by Designcited by 18
  34. 2021CubeLearn: End-to-end Learning for Human Motion Recognition from Raw mmWave Radar Signalscited by 18
  35. 2023Infeasible Deterministic, Stochastic, and Variance-Reduction Algorithms for Optimization under Orthogonality Constraintscited by 18
  36. 2023Robustifying State-space Models for Long Sequences via Approximate Diagonalizationcited by 18
  37. 2024State-Free Inference of State-Space Models: The Transfer Function Approachcited by 18
  38. 2021MC-LSTM: Mass-Conserving LSTMcited by 17

50 most cited of 294 All 294 (JSON)