mlsanityData to 14 Oct 2026

Orthogonal Gated Recurrent Unit with Neumann-Cayley Transformation

arXiv 2208.06496 ↗First posted 12 Aug 2022cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

In recent years, using orthogonal matrices has been shown to be a promising approach in improving Recurrent Neural Networks (RNNs) with training, stability, and convergence, particularly, to control gradients. While Gated Recurrent Unit (GRU) and Long Short Term Memory (LSTM) architectures address the vanishing gradient problem by using a variety of gates and memory cells, they are still prone to the exploding gradient problem. In this work, we analyze the gradients in GRU and propose the usage of orthogonal matrices to prevent exploding gradient problems and enhance long-term memory. We study where to use orthogonal matrices and we propose a Neumann series-based Scaled Cayley transformation for training orthogonal matrices in GRU, which we call Neumann-Cayley Orthogonal GRU, or simply NC-GRU. We present detailed experiments of our model on several synthetic and real-world tasks, which show that NC-GRU significantly outperforms GRU as well as several other RNNs.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 24 of its 37 references that are in the map. First posted on arXiv 12 Aug 2022.

Builds on

24 of 37 references.

References · 24

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  3. 2016Layer Normalizationcited by 5,669
  4. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  5. 2018Group Normalizationcited by 1,524
  6. 2018An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modelingcited by 1,476
  7. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,198
  8. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  9. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 543
  10. 2015Unitary Evolution Recurrent Neural Networkscited by 294
  11. 2019Understanding and Improving Layer Normalizationcited by 217
  12. 2017Regularizing and Optimizing LSTM Language Modelscited by 205
12 more
  1. 2019Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Groupcited by 134
  2. 2016Full-Capacity Unitary Recurrent Neural Networkscited by 94
  3. 2017On orthogonality and learning recurrent networks with long term dependenciescited by 83
  4. 2017Orthogonal Recurrent Neural Networks with Scaled Cayley Transformcited by 83
  5. 2016Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNscited by 65
  6. 2016Efficient Orthogonal Parametrisation of Recurrent Neural Networks Using Householder Reflectionscited by 57
  7. 2016Recurrent Orthogonal Networks and Long-Memory Taskscited by 38
  8. 2016Recurrent Batch Normalizationcited by 36
  9. 2017Gated Orthogonal Recurrent Units: On Learning to Forgetcited by 31
  10. 2016Learning Unitary Operators with Help From u(n)cited by 15
  11. 2018Complex Unitary Recurrent Neural Networks using Scaled Cayley Transformcited by 12
  12. 2019Eigenvalue Normalized Recurrent Neural Networks for Short Term Memorycited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 1; 2024: 0; 2025: 1; 2026: 1.

4 papers.

  1. 2026Cold-Start Forecasting of New Product Life-Cycles via Conditional Diffusion Modelscited by 1
  2. 2022Breaking Time Invariance: Assorted-Time Normalization for RNNscited by 0
  3. 2023Mathematics-assisted directed evolution and protein engineeringcited by 0
  4. 2025Compact Recurrent Transformer with Persistent Memorycited by 0