mlsanityData to 7 Oct 2026

Orthogonal Recurrent Neural Networks with Scaled Cayley Transform

arXiv 1707.09520 ↗First posted 29 Jul 2017stat.ML cs.LGcited by 82

Not on the map. Why?

Abstract (arXiv)

Recurrent Neural Networks (RNNs) are designed to handle sequential data but suffer from vanishing or exploding gradients. Recent work on Unitary Recurrent Neural Networks (uRNNs) have been used to address this issue and in some cases, exceed the capabilities of Long Short-Term Memory networks (LSTMs). We propose a simpler and novel update scheme to maintain orthogonal recurrent weight matrices without using complex valued matrices. This is done by parametrizing with a skew-symmetric matrix using the Cayley transform. Such a parametrization is unable to represent matrices with negative one eigenvalues, but this limitation is overcome by scaling the recurrent weight matrix by a diagonal matrix consisting of ones and negative ones. The proposed training scheme involves a straightforward gradient calculation and update step. In several experiments, the proposed scaled Cayley orthogonal recurrent neural network (scoRNN) achieves superior results with fewer trainable parameters than other unitary RNNs.

Key facts. Cited by 82 papers in the map (data through 7 Oct 2026). First posted on arXiv 29 Jul 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 19; 2022: 7; 2023: 10; 2024: 15; 2025: 9; 2026: 22.

82 papers.

Sort:

By area:

  1. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
  2. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  3. 2021Orthogonalizing Convolutional Layers with the Cayley Transformcited by 105
  4. 2021On the difficulty of learning chaotic dynamics with RNNscited by 64
  5. 2021UnICORNN: A recurrent model for learning very long time dependenciescited by 56
  6. 2021Heavy Ball Neural Ordinary Differential Equationscited by 42
  7. 2021Long Expressive Memory for Sequence Modelingcited by 41
  8. 2021Fast and accurate optimization on the orthogonal manifold without retractioncited by 35
  9. 2023Deep networks for system identification: a Surveycited by 35
  10. 2021Pay attention to your loss: understanding misconceptions about 1-Lipschitz neural networkscited by 30
  11. 2022projUNN: efficient method for training deep networks with unitary matricescited by 27
  12. 2024Recurrent neural networks: vanishing and exploding gradients are not the end of the storycited by 25
38 more
  1. 2024VkD: Improving Knowledge Distillation using Orthogonal Projectionscited by 24
  2. 2021Hamiltonian Deep Neural Networks Guaranteeing Non-vanishing Gradients by Designcited by 18
  3. 2021MC-LSTM: Mass-Conserving LSTMcited by 17
  4. 2021Convolutional Normalization: Improving Deep Convolutional Network Robustness and Trainingcited by 15
  5. 2021Distributed neural network control with dependability guarantees: a compositional port-Hamiltonian approachcited by 12
  6. 2021How Does Momentum Benefit Deep Neural Networks Architecture Design? A Few Case Studiescited by 10
  7. 2022Momentum Stiefel Optimizer, with Applications to Suitably-Orthogonal Attention, and Optimal Transportcited by 10
  8. 2024Group and Shuffle: Efficient Structured Orthogonal Parametrizationcited by 10
  9. 2023Lipschitz-bounded 1D convolutional neural networks using the Cayley transform and the controllability Gramiancited by 9
  10. 2024Monotone, Bi-Lipschitz, and Polyak-Lojasiewicz Networkscited by 9
  11. 2025Quantum-PEFT: Ultra parameter-efficient fine-tuningcited by 9
  12. 2023Scalable Neural Network Kernelscited by 8
  13. 2024Unitary convolutions for learning on graphs and groupscited by 8
  14. 2025ButterflyQuant: Ultra-low-bit LLM Quantization through Learnable Orthogonal Butterfly Transformscited by 8
  15. 2021Scaling-up Diverse Orthogonal Convolutional Networks with a Paraunitary Frameworkcited by 7
  16. 2022NCVX: A General-Purpose Optimization Solver for Constrained Machine and Deep Learningcited by 7
  17. 2024Neural Gaussian Scale-Space Fieldscited by 6
  18. 2024RotRNN: Modelling Long Sequences with Rotationscited by 6
  19. 2021Coordinate descent on the orthogonal group for recurrent neural network trainingcited by 5
  20. 2022Feedback Gradient Descent: Efficient and Stable Optimization with Orthogonality for DNNscited by 5
  21. 2022Random orthogonal additive filters: a solution to the vanishing/exploding gradient of deep neural networkscited by 5
  22. 2023Gradient Flossing: Improving Gradient Descent through Dynamic Control of Jacobianscited by 5
  23. 2024Graph Unitary Message Passingcited by 5
  24. 2025Fixed-Point RNNs: Interpolating from Diagonal to Densecited by 5
  25. 2026Pretraining Recurrent Networks without Recurrencecited by 5
  26. 2022Orthogonal Gated Recurrent Unit with Neumann-Cayley Transformationcited by 4
  27. 2021A Differential Geometry Perspective on Orthogonal Recurrent Modelscited by 3
  28. 2021Target Propagation via Regularized Inversioncited by 3
  29. 2025Efficient Optimization with Orthogonality Constraint: a Randomized Riemannian Submanifold Methodcited by 3
  30. 2026go-mHC: Direct Parameterization of Manifold-Constrained Hyper-Connections via Generalized Orthostochastic Matricescited by 3
  31. 2021Deep Unitary Convolutional Neural Networkscited by 2
  32. 2021Oscillatory Fourier Neural Network: A Compact and Efficient Architecture for Sequential Processingcited by 2
  33. 2022Givens Coordinate Descent Methods for Rotation Matrix Learning in Trainable Embedding Indexescited by 2
  34. 2023Fading memory as inductive bias in residual recurrent networkscited by 2
  35. 2023Leveraging Low-Rank and Sparse Recurrent Connectivity for Robust Closed-Loop Controlcited by 2
  36. 2024LipKernel: Lipschitz-Bounded Convolutional Neural Networks via Dissipative Layerscited by 2
  37. 2026Pion: A Spectrum-Preserving Optimizer via Orthogonal Equivalence Transformationcited by 2
  38. 2026MuonSSM: Orthogonalizing State Space Models for Sequence Modelingcited by 2

50 most cited of 82 All 82 (JSON)