mlsanityData to 7 Oct 2026

On orthogonality and learning recurrent networks with long term dependencies

arXiv 1702.00071 ↗First posted 31 Jan 2017cs.LG cs.NEcited by 82

Not on the map. Why?

Abstract (arXiv)

It is well known that it is challenging to train deep neural networks and recurrent neural networks for tasks that exhibit long term dependencies. The vanishing or exploding gradient problem is a well known issue associated with these challenges. One approach to addressing vanishing and exploding gradients is to use either soft or hard constraints on weight matrices so as to encourage or enforce orthogonality. Orthogonal matrices preserve gradient norm during backpropagation and may therefore be a desirable property. This paper explores issues with optimization convergence, speed and gradient stability when encouraging or enforcing orthogonality. To perform this analysis, we propose a weight matrix factorization and parameterization strategy through which we can bound matrix norms and therein control the degree of expansivity induced during backpropagation. We find that hard constraints on orthogonality can negatively affect the speed of convergence and model performance.

Key facts. Cited by 82 papers in the map (data through 7 Oct 2026). First posted on arXiv 31 Jan 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 22; 2022: 14; 2023: 14; 2024: 11; 2025: 12; 2026: 9.

82 papers.

Sort:

By area:

  1. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  2. 2021Orthogonalizing Convolutional Layers with the Cayley Transformcited by 105
  3. 2021Heavy Ball Neural Ordinary Differential Equationscited by 42
  4. 2023Initializing Models with Larger Onescited by 39
  5. 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
  6. 2022The Principle of Diversity: Training Stronger Vision Transformers Calls for Reducing All Levels of Redundancycited by 35
  7. 2023Deep networks for system identification: a Surveycited by 35
  8. 2023Auxiliary Losses for Learning Generalizable Concept-based Modelscited by 35
  9. 2021Decentralized Riemannian Gradient Descent on the Stiefel Manifoldcited by 33
  10. 2021Deep Learning Methods for Vessel Trajectory Prediction based on Recurrent Neural Networkscited by 32
  11. 2024Recurrent neural networks: vanishing and exploding gradients are not the end of the storycited by 25
  12. 2023A Survey of Geometric Optimization for Deep Learning: From Euclidean Space to Riemannian Manifoldcited by 21
38 more
  1. 2021Improving Molecular Graph Neural Network Explainability with Orthonormalization and Induced Sparsitycited by 20
  2. 2021Learning Connectivity with Graph Convolutional Networks for Skeleton-based Action Recognitioncited by 20
  3. 2021Orthogonal Graph Neural Networkscited by 19
  4. 2021Hamiltonian Deep Neural Networks Guaranteeing Non-vanishing Gradients by Designcited by 18
  5. 2021Convolutional Normalization: Improving Deep Convolutional Network Robustness and Trainingcited by 15
  6. 2021Distributed neural network control with dependability guarantees: a compositional port-Hamiltonian approachcited by 12
  7. 2021Decentralized Optimization Over the Stiefel Manifold by an Approximate Augmented Lagrangian Functioncited by 11
  8. 2021How Does Momentum Benefit Deep Neural Networks Architecture Design? A Few Case Studiescited by 10
  9. 2022Momentum Stiefel Optimizer, with Applications to Suitably-Orthogonal Attention, and Optimal Transportcited by 10
  10. 2024Group and Shuffle: Efficient Structured Orthogonal Parametrizationcited by 10
  11. 2025Riemannian Optimization for LoRA on the Stiefel Manifoldcited by 10
  12. 2025Long-tailed Recognition with Model Rebalancingcited by 10
  13. 2021On the Local Linear Rate of Consensus on the Stiefel Manifoldcited by 8
  14. 2023Towards Training Without Depth Limits: Batch Normalization Without Gradient Explosioncited by 8
  15. 2021Scaling-up Diverse Orthogonal Convolutional Networks with a Paraunitary Frameworkcited by 7
  16. 2023Decentralized Riemannian Conjugate Gradient Method on the Stiefel Manifoldcited by 7
  17. 2024Generalizing Knowledge Graph Embedding with Universal Orthogonal Parameterizationcited by 7
  18. 2021Stable Long-Term Recurrent Video Super-Resolutioncited by 6
  19. 2023Persistent learning signals and working memory without continuous attractorscited by 6
  20. 2025Over-squashing in Spatiotemporal Graph Neural Networkscited by 6
  21. 2022Feedback Gradient Descent: Efficient and Stable Optimization with Orthogonality for DNNscited by 5
  22. 2022Random orthogonal additive filters: a solution to the vanishing/exploding gradient of deep neural networkscited by 5
  23. 2022Gated Recurrent Neural Networks with Weighted Time-Delay Feedbackcited by 5
  24. 2023Leveraging Neural Koopman Operators to Learn Continuous Representations of Dynamical Systems from Scarce Datacited by 5
  25. 2026Pretraining Recurrent Networks without Recurrencecited by 5
  26. 2021Better Training using Weight-Constrained Stochastic Dynamicscited by 4
  27. 2022Entangled Residual Mappingscited by 4
  28. 2022Orthogonal Gated Recurrent Unit with Neumann-Cayley Transformationcited by 4
  29. 2021A Differential Geometry Perspective on Orthogonal Recurrent Modelscited by 3
  30. 2021Preprint: Norm Loss: An efficient yet effective regularization method for deep neural networkscited by 3
  31. 2021Exploiting a Zoo of Checkpoints for Unseen Taskscited by 3
  32. 2022Recency Dropout for Recurrent Recommender Systemscited by 3
  33. 2022Lyapunov-Guided Representation of Recurrent Neural Network Performancecited by 3
  34. 2023Stabilising and accelerating light gated recurrent units for automatic speech recognitioncited by 3
  35. 2023Towards Better Orthogonality Regularization with Disentangled Norm in Training Deep CNNscited by 3
  36. 2023The Deep Arbitrary Polynomial Chaos Neural Network or how Deep Artificial Neural Networks could benefit from Data-Driven Homogeneous Chaos Theorycited by 3
  37. 2021Deep Unitary Convolutional Neural Networkscited by 2
  38. 2022A Variance-Reduced Stochastic Gradient Tracking Algorithm for Decentralized Optimization with Orthogonality Constraintscited by 2

50 most cited of 82 All 82 (JSON)