# Gated recurrent units viewed through the lens of continuous time dynamical systems

arXiv: https://arxiv.org/abs/1906.01005 · HTML: https://mlsanity.com/paper/1906.01005/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Jun 2019.

## Built on by (most cited first, 18 of 18)

- [Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers](https://mlsanity.com/paper/2110.13985/) (2021)
- [Attentive Neural Controlled Differential Equations for Time-series Classification and Forecasting](https://mlsanity.com/paper/2109.01876/) (2021)
- [Zero-shot forecasting of chaotic systems](https://mlsanity.com/paper/2409.15771/) (2024)
- [Efficient recurrent architectures through activity sparsity and sparse back-propagation through time](https://mlsanity.com/paper/2206.06178/) (2022)
- [EXIT: Extrapolation and Interpolation-based Neural Controlled Differential Equations for Time-series Classification and Forecasting](https://mlsanity.com/paper/2204.08771/) (2022)
- [Reverse engineering recurrent neural networks with Jacobian switching linear dynamical systems](https://mlsanity.com/paper/2111.01256/) (2021)
- [Context parroting: A simple but tough-to-beat baseline for foundation models in scientific machine learning](https://mlsanity.com/paper/2505.11349/) (2025)
- [Reduced-Order Autodifferentiable Ensemble Kalman Filters](https://mlsanity.com/paper/2301.11961/) (2023)
- [Deep Efficient Continuous Manifold Learning for Time Series Modeling](https://mlsanity.com/paper/2112.03379/) (2021)
- [Learnable Path in Neural Controlled Differential Equations](https://mlsanity.com/paper/2301.04333/) (2023)
- [Persistent learning signals and working memory without continuous attractors](https://mlsanity.com/paper/2308.12585/) (2023)
- [Parallel Training of GRU Networks with a Multi-Grid Solver for Long Sequences](https://mlsanity.com/paper/2203.04738/) (2022)
- [Neural timescales from a computational perspective](https://mlsanity.com/paper/2409.02684/) (2024)
- [Trainability, Expressivity and Interpretability in Gated Neural ODEs](https://mlsanity.com/paper/2307.06398/) (2023)
- [Sufficient conditions for offline reactivation in recurrent neural networks](https://mlsanity.com/paper/2505.17003/) (2025)
- [Multiscale autonomous forecasting of plasma systems' dynamics using neural networks](https://mlsanity.com/paper/2502.11203/) (2025)
- [Frequency-Domain Neural ODEs for Modeling Non-Linear Dynamical Systems](https://mlsanity.com/paper/2606.22075/) (2026)
- [Divisive Normalization Shapes Low-Rank Slow Manifolds for Continuous Working Memory](https://mlsanity.com/paper/2608.01947/) (2026)

## Abstract (arXiv)

Gated recurrent units (GRUs) are specialized memory elements for building recurrent neural networks. Despite their incredible success on various tasks, including extracting dynamics underlying neural data, little is understood about the specific dynamics representable in a GRU network. As a result, it is both difficult to know a priori how successful a GRU network will perform on a given task, and also their capacity to mimic the underlying behavior of their biological counterparts. Using a continuous time analysis, we gain intuition on the inner workings of GRU networks. We restrict our presentation to low dimensions, allowing for a comprehensive visualization. We found a surprisingly rich repertoire of dynamical features that includes stable limit cycles (nonlinear oscillations), multi-stable dynamics with various topologies, and homoclinic bifurcations. At the same time we were unable to train GRU networks to produce continuous attractors, which are hypothesized to exist in biological neural networks. We contextualize the usefulness of different kinds of observed dynamics and support our claims experimentally.