# Non-normal Recurrent Neural Network (nnRNN): learning long time dependencies while improving expressivity with transient dynamics

arXiv: https://arxiv.org/abs/1905.12080 · HTML: https://mlsanity.com/paper/1905.12080/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 May 2019.

## Built on by (most cited first, 29 of 29)

- [On the difficulty of learning chaotic dynamics with RNNs](https://mlsanity.com/paper/2110.07238/) (2021)
- [UnICORNN: A recurrent model for learning very long time dependencies](https://mlsanity.com/paper/2103.05487/) (2021)
- [Long Expressive Memory for Sequence Modeling](https://mlsanity.com/paper/2110.04744/) (2021)
- [Physics-Informed Machine Learning for Modeling and Control of Dynamical Systems](https://mlsanity.com/paper/2306.13867/) (2023)
- [Robustifying State-space Models for Long Sequences via Approximate Diagonalization](https://mlsanity.com/paper/2310.01698/) (2023)
- [Temporally Consistent Koopman Autoencoders for Forecasting Dynamical Systems](https://mlsanity.com/paper/2403.12335/) (2024)
- [Neural oscillators for generalization of physics-informed machine learning](https://mlsanity.com/paper/2308.08989/) (2023)
- [Think Shallow, Solve Deep: Controlling Recurrent Dynamics for Reliable Test-Time Depth](https://mlsanity.com/paper/2608.18222/) (2026)
- [Slower is Better: Revisiting the Forgetting Mechanism in LSTM for Slower Information Decay](https://mlsanity.com/paper/2105.05944/) (2021)
- [Koopman Invertible Autoencoder: Leveraging Forward and Backward Dynamics for Temporal Modeling](https://mlsanity.com/paper/2309.10291/) (2023)
- [Random orthogonal additive filters: a solution to the vanishing/exploding gradient of deep neural networks](https://mlsanity.com/paper/2210.01245/) (2022)
- [Gated Recurrent Neural Networks with Weighted Time-Delay Feedback](https://mlsanity.com/paper/2212.00228/) (2022)
- [A Differential Geometry Perspective on Orthogonal Recurrent Models](https://mlsanity.com/paper/2102.09589/) (2021)
- [Lyapunov-Guided Representation of Recurrent Neural Network Performance](https://mlsanity.com/paper/2204.04876/) (2022)
- [State space models can express n-gram languages](https://mlsanity.com/paper/2306.17184/) (2023)
- [Eigenvalue initialisation and regularisation for Koopman autoencoders](https://mlsanity.com/paper/2212.12086/) (2022)
- [Fading memory as inductive bias in residual recurrent networks](https://mlsanity.com/paper/2307.14823/) (2023)
- [Dynamical symmetries in the fluctuation-driven regime: an application of Noether's theorem to noisy dynamical systems](https://mlsanity.com/paper/2504.09761/) (2025)
- [Time-Scale Coupling Between States and Parameters in Recurrent Neural Networks](https://mlsanity.com/paper/2508.12121/) (2025)
- [Quantized Approximately Orthogonal Recurrent Neural Networks](https://mlsanity.com/paper/2402.04012/) (2024)
- [Non-equilibrium active noise enhances generative memory in diffusion models](https://mlsanity.com/paper/2411.07233/) (2024)
- [When Recursive Models Finish Computing](https://mlsanity.com/paper/2609.26487/) (2026)
- [Adaptive-saturated RNN: Remember more with less instability](https://mlsanity.com/paper/2304.11790/) (2023)
- [Extending Memory for Language Modelling](https://mlsanity.com/paper/2305.11462/) (2023)
- [Stabilizing RNN Gradients through Pre-training](https://mlsanity.com/paper/2308.12075/) (2023)
- [Unified Implementations of Recurrent Neural Networks in Multiple Deep Learning Frameworks](https://mlsanity.com/paper/2510.21252/) (2025)
- [DAGGER: Gradient-Free Construction of Transiently Amplifying Networks under Hard Connectivity Constraints](https://mlsanity.com/paper/2606.01227/) (2026)
- [Finite-Horizon Fisher Memory in Two-Sided Power-Bounded Recurrent Systems](https://mlsanity.com/paper/2609.39800/) (2026)
- [Riding the Fixed-Point Curve: Image Classification in the Continuous Thought Machine](https://mlsanity.com/paper/2610.19220/) (2026)

## Abstract (arXiv)

A recent strategy to circumvent the exploding and vanishing gradient problem in RNNs, and to allow the stable propagation of signals over long time scales, is to constrain recurrent connectivity matrices to be orthogonal or unitary. This ensures eigenvalues with unit norm and thus stable dynamics and training. However this comes at the cost of reduced expressivity due to the limited variety of orthogonal transformations. We propose a novel connectivity structure based on the Schur decomposition and a splitting of the Schur form into normal and non-normal parts. This allows to parametrize matrices with unit-norm eigenspectra without orthogonality constraints on eigenbases. The resulting architecture ensures access to a larger space of spectrally constrained matrices, of which orthogonal matrices are a subset. This crucial difference retains the stability advantages and training speed of orthogonal RNNs while enhancing expressivity, especially on tasks that require computations over ongoing input sequences.