mlsanityData to 14 Oct 2026

Recurrent Orthogonal Networks and Long-Memory Tasks

arXiv 1602.06662 ↗First posted 22 Feb 2016cs.NE cs.AI cs.LGcited by 38

Not on the map. Why?

Abstract (arXiv)

Although RNNs have been shown to be powerful tools for processing sequential data, finding architectures or optimization strategies that allow them to model very long term dependencies is still an active area of research. In this work, we carefully analyze two synthetic datasets originally outlined in (Hochreiter and Schmidhuber, 1997) which are used to evaluate the ability of RNNs to store information over many time steps. We explicitly construct RNN solutions to these problems, and using these constructions, illuminate both the problems themselves and the way in which RNNs store different types of information in their hidden states. These constructions furthermore explain the success of recent methods that specify unitary initializations or constraints on the transition matrices.

Key facts. Cited by 38 papers in the map (data through 14 Oct 2026). First posted on arXiv 22 Feb 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 11; 2022: 5; 2023: 5; 2024: 4; 2025: 6; 2026: 7.

38 papers.

Sort:

By area:

  1. 2021On the difficulty of learning chaotic dynamics with RNNscited by 67
  2. 2021UnICORNN: A recurrent model for learning very long time dependenciescited by 56
  3. 2025On Vanishing Gradients, Over-Smoothing, and Over-Squashing in GNNs: Bridging Recurrent and Graph Learningcited by 43
  4. 2021Long Expressive Memory for Sequence Modelingcited by 41
  5. 2021Deep Learning Methods for Vessel Trajectory Prediction based on Recurrent Neural Networkscited by 32
  6. 2022projUNN: efficient method for training deep networks with unitary matricescited by 27
  7. 2023Traveling Waves Encode the Recent Past and Enhance Sequence Learningcited by 16
  8. 2024Rough Transformers: Lightweight and Continuous Time Series Modelling through Signature Patchingcited by 16
  9. 2021HeterPS: Distributed Deep Learning With Reinforcement Learning Based Scheduling in Heterogeneous Environmentscited by 15
  10. 2021How Does Momentum Benefit Deep Neural Networks Architecture Design? A Few Case Studiescited by 10
  11. 2023Neural oscillators for generalization of physics-informed machine learningcited by 9
  12. 2023Towards Training Without Depth Limits: Batch Normalization Without Gradient Explosioncited by 8
26 more
  1. 2024Unitary convolutions for learning on graphs and groupscited by 8
  2. 2021Scaling-up Diverse Orthogonal Convolutional Networks with a Paraunitary Frameworkcited by 7
  3. 2022NCVX: A General-Purpose Optimization Solver for Constrained Machine and Deep Learningcited by 7
  4. 2021Coordinate descent on the orthogonal group for recurrent neural network trainingcited by 5
  5. 2025Uncovering the Computational Roles of Nonlinearity in Sequence Modeling Using Almost-Linear RNNscited by 5
  6. 2022Orthogonal Gated Recurrent Unit with Neumann-Cayley Transformationcited by 4
  7. 2025Counterfactual reasoning: an analysis of in-context emergencecited by 4
  8. 2026go-mHC: Direct Parameterization of Manifold-Constrained Hyper-Connections via Generalized Orthostochastic Matricescited by 4
  9. 2021A Differential Geometry Perspective on Orthogonal Recurrent Modelscited by 3
  10. 2026MuonSSM: Orthogonalizing State Space Models for Sequence Modelingcited by 3
  11. 2021Emergent behavior and neural dynamics in artificial agents tracking turbulent plumescited by 2
  12. 2025Hybrid Quantum Neural Networks with Amplitude Encoding: Advancing Recovery Rate Predictionscited by 2
  13. 2025Revisiting Glorot Initialization for Long-Range Linear Recurrencescited by 2
  14. 2026Spontaneous symmetry breaking and Goldstone modes for deep information propagationcited by 2
  15. 2021Learning future terrorist targets through temporal meta-graphscited by 1
  16. 2022Euler State Networks: Non-dissipative Reservoir Computingcited by 1
  17. 2022On the biological plausibility of orthogonal initialisation for solving gradient instability in deep neural networkscited by 1
  18. 2024Quantized Approximately Orthogonal Recurrent Neural Networkscited by 1
  19. 2024Oja's plasticity rule overcomes several challenges of training neural networks under biological constraintscited by 1
  20. 2026SAFE Quantum Machine Learning with Variational Quantum Classifierscited by 1
  21. 2023Adaptive-saturated RNN: Remember more with less instabilitycited by 0
  22. 2023Stabilizing RNN Gradients through Pre-trainingcited by 0
  23. 2025Can Local Representation Alignment RNNs Solve Temporal Tasks?cited by 0
  24. 2026On Training in Imaginationcited by 0
  25. 2026Learning in Low-Dimensional Subspaces: Orthogonal Bottlenecks for Reinforcement Learningcited by 0
  26. 2026Stable Transformers for Graph Generationcited by 0