# Lyapunov-Guided Representation of Recurrent Neural Network Performance

arXiv: https://arxiv.org/abs/2204.04876 · HTML: https://mlsanity.com/paper/2204.04876/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 70 references that are in the map.
- First posted on arXiv 11 Apr 2022.

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://mlsanity.com/paper/1803.01271/) (2018)
- [Deep learning for universal linear embeddings of nonlinear dynamics](https://mlsanity.com/paper/1712.09707/) (2017)
- [Exponential expressivity in deep neural networks through transient chaos](https://mlsanity.com/paper/1606.05340/) (2016)
- [DRAW: A Recurrent Neural Network For Image Generation](https://mlsanity.com/paper/1502.04623/) (2015)
- [Visualizing and Understanding Recurrent Networks](https://mlsanity.com/paper/1506.02078/) (2015)
- [Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation](https://mlsanity.com/paper/1902.04760/) (2019)
- [Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice](https://mlsanity.com/paper/1711.04735/) (2017)
- [Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data](https://mlsanity.com/paper/2002.06716/) (2020)
- [Forecasting Sequential Data using Consistent Koopman Autoencoders](https://mlsanity.com/paper/2003.02236/) (2020)
- [AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks](https://mlsanity.com/paper/1902.09689/) (2019)
- [Full-Capacity Unitary Recurrent Neural Networks](https://mlsanity.com/paper/1611.00035/) (2016)
- [PREDICT & CLUSTER: Unsupervised Skeleton Based Action Recognition](https://mlsanity.com/paper/1911.12409/) (2019)
- [Lipschitz Recurrent Neural Networks](https://mlsanity.com/paper/2006.12070/) (2020)
- [On orthogonality and learning recurrent networks with long term dependencies](https://mlsanity.com/paper/1702.00071/) (2017)
- [Coupled Oscillatory Recurrent Neural Network (coRNN): An accurate and (gradient) stable architecture for learning long time dependencies](https://mlsanity.com/paper/2010.00951/) (2020)
- [Deep Dynamical Modeling and Control of Unsteady Fluid Flows](https://mlsanity.com/paper/1805.07472/) (2018)
- [Physics-informed Autoencoders for Lyapunov-stable Fluid Flow Prediction](https://mlsanity.com/paper/1905.10866/) (2019)
- [On the difficulty of learning chaotic dynamics with RNNs](https://mlsanity.com/paper/2110.07238/) (2021)
- [Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNs](https://mlsanity.com/paper/1612.05231/) (2016)
- [Universality and individuality in neural dynamics across large populations of recurrent networks](https://mlsanity.com/paper/1907.08549/) (2019)
- [Efficient Orthogonal Parametrisation of Recurrent Neural Networks Using Householder Reflections](https://mlsanity.com/paper/1612.00188/) (2016)
- [From Fourier to Koopman: Spectral Methods for Long-term Time Series Prediction](https://mlsanity.com/paper/2004.00574/) (2020)
- [Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks](https://mlsanity.com/paper/1806.05394/) (2018)
- [Noisy Recurrent Neural Networks](https://mlsanity.com/paper/2102.04877/) (2021)
- [Long Expressive Memory for Sequence Modeling](https://mlsanity.com/paper/2110.04744/) (2021)
- [Lyapunov spectra of chaotic recurrent neural networks](https://mlsanity.com/paper/2006.02427/) (2020)

## Built on by (most cited first, 3 of 3)

- [Neural Tangent Knowledge Distillation for Optical Convolutional Networks](https://mlsanity.com/paper/2508.08421/) (2025)
- [Hyperpruning: Efficient Search through Pruned Variants of Recurrent Neural Networks Leveraging Lyapunov Spectrum](https://mlsanity.com/paper/2506.07975/) (2025)
- [2ndMatch: Finetuning Pruned Diffusion Models via Second-Order Jacobian Matching](https://mlsanity.com/paper/2506.05398/) (2025)

## Abstract (arXiv)

Recurrent Neural Networks (RNN) are ubiquitous computing systems for sequences and multivariate time series data. While several robust architectures of RNN are known, it is unclear how to relate RNN initialization, architecture, and other hyperparameters with accuracy for a given task. In this work, we propose to treat RNN as dynamical systems and to correlate hyperparameters with accuracy through Lyapunov spectral analysis, a methodology specifically designed for nonlinear dynamical systems. To address the fact that RNN features go beyond the existing Lyapunov spectral analysis, we propose to infer relevant features from the Lyapunov spectrum with an Autoencoder and an embedding of its latent representation (AeLLE). Our studies of various RNN architectures show that AeLLE successfully correlates RNN Lyapunov spectrum with accuracy. Furthermore, the latent representation learned by AeLLE is generalizable to novel inputs from the same task and is formed early in the process of RNN training. The latter property allows for the prediction of the accuracy to which RNN would converge when training is complete. We conclude that representation of RNN through Lyapunov spectrum along with AeLLE provides a novel method for organization and interpretation of variants of RNN architectures.