mlsanityData to 7 Oct 2026

Lipschitz Recurrent Neural Networks

arXiv 2006.12070 ↗First posted 22 Jun 2020cs.LG math.DS stat.MLcited by 88

Not on the map. Why?

Abstract (arXiv)

Viewing recurrent neural networks (RNNs) as continuous-time dynamical systems, we propose a recurrent unit that describes the hidden state's evolution with two parts: a well-understood linear component plus a Lipschitz nonlinearity. This particular functional form facilitates stability analysis of the long-term behavior of the recurrent unit using tools from nonlinear systems theory. In turn, this enables architectural design decisions before experimentation. Sufficient conditions for global stability of the recurrent unit are obtained, motivating a novel scheme for constructing hidden-to-hidden matrices. Our experiments demonstrate that the Lipschitz RNN can outperform existing recurrent units on a range of benchmark tasks, including computer vision, language modeling and speech prediction tasks. Finally, through Hessian-based analysis we demonstrate that our Lipschitz recurrent unit is more robust with respect to input and parameter perturbations as compared to other continuous-time RNNs.

Key facts. Cited by 88 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 20; 2022: 13; 2023: 18; 2024: 16; 2025: 10; 2026: 11.

88 papers.

Sort:

By area:

  1. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  2. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  3. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 607
  4. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 418
  5. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
  6. 2022Liquid Structural State-Space Modelscited by 134
  7. 2021FlexConv: Continuous Kernel Convolutions with Differentiable Kernel Sizescited by 66
  8. 2021Closed-form Continuous-time Neural Modelscited by 65
  9. 2021On the difficulty of learning chaotic dynamics with RNNscited by 64
  10. 2021UnICORNN: A recurrent model for learning very long time dependenciescited by 56
  11. 2023Generalized Teacher Forcing for Learning Chaotic Dynamicscited by 55
  12. 2021Noisy Recurrent Neural Networkscited by 42
38 more
  1. 2021Heavy Ball Neural Ordinary Differential Equationscited by 42
  2. 2024Utilizing Image Transforms and Diffusion Models for Generative Modeling of Short and Long Time Seriescited by 42
  3. 2021Long Expressive Memory for Sequence Modelingcited by 41
  4. 2021Pay attention to your loss: understanding misconceptions about 1-Lipschitz neural networkscited by 30
  5. 2022Modeling Multivariate Biosignals With Graph Neural Networks and Structured State Space Modelscited by 29
  6. 2022Learning continuous models for continuous physicscited by 28
  7. 2023Physics-Informed Machine Learning for Modeling and Control of Dynamical Systemscited by 28
  8. 2022projUNN: efficient method for training deep networks with unitary matricescited by 27
  9. 2023Some Fundamental Aspects about Lipschitz Continuity of Neural Networkscited by 25
  10. 2021Framing RNN as a kernel method: A neural ODE approachcited by 23
  11. 2021Causal Navigation by Continuous-time Neural Networkscited by 23
  12. 2023Sequence Modeling with Multiresolution Convolutional Memorycited by 19
  13. 2023Robustifying State-space Models for Long Sequences via Approximate Diagonalizationcited by 18
  14. 2023Estimating Treatment Effects from Irregular Time Series Observations with Hidden Confounderscited by 17
  15. 2023Traveling Waves Encode the Recent Past and Enhance Sequence Learningcited by 16
  16. 2024Rough Transformers: Lightweight and Continuous Time Series Modelling through Signature Patchingcited by 16
  17. 2024Tuning Frequency Bias of State Space Modelscited by 16
  18. 2021Stateful ODE-Nets using Basis Function Expansionscited by 14
  19. 2021Sparse Flows: Pruning Continuous-depth Modelscited by 13
  20. 2021An Operator Theoretic Approach for Analyzing Sequence Neural Networkscited by 12
  21. 2022Flipped Classroom: Effective Teaching for Time Series Forecastingcited by 11
  22. 2022Path Development Network with Finite-dimensional Lie Group Representationcited by 8
  23. 2021RNNs of RNNs: Recursive Construction of Stable Assemblies of Recurrent Neural Networkscited by 7
  24. 2024Sequential Disentanglement by Extracting Static Information From A Single Sequence Elementcited by 7
  25. 2025Block-Biased Mamba for Long-Range Sequence Processingcited by 7
  26. 2025Predictability Enables Parallelization of Nonlinear State Space Modelscited by 7
  27. 2023Persistent learning signals and working memory without continuous attractorscited by 6
  28. 2024A Diagonal Structured State Space Model on Loihi 2 for Efficient Streaming Sequence Processingcited by 6
  29. 2022Random orthogonal additive filters: a solution to the vanishing/exploding gradient of deep neural networkscited by 5
  30. 2022Gated Recurrent Neural Networks with Weighted Time-Delay Feedbackcited by 5
  31. 2023Gradient Flossing: Improving Gradient Descent through Dynamic Control of Jacobianscited by 5
  32. 2024Short-Long Convolutions Help Hardware-Efficient Linear Attention to Focus on Long Sequencescited by 5
  33. 2023Neural Differential Recurrent Neural Network with Adaptive Time Stepscited by 4
  34. 2023A Unified Framework for Generative Data Augmentation: A Comprehensive Surveycited by 4
  35. 2023Delayed Memory Unit: Modelling Temporal Dependency Through Delay Gatecited by 4
  36. 2024LongVQ: Long Sequence Modeling with Vector Quantization on Structured Memorycited by 4
  37. 2024Unconditional stability of a recurrent neural circuit implementing divisive normalizationcited by 4
  38. 2021A Differential Geometry Perspective on Orthogonal Recurrent Modelscited by 3

50 most cited of 88 All 88 (JSON)