mlsanityData to 7 Oct 2026

UnICORNN: A recurrent model for learning very long time dependencies

arXiv 2103.05487 ↗First posted 9 Mar 2021cs.LG math.DS stat.MLcited by 56

Not on the map. Why?

Abstract (arXiv)

The design of recurrent neural networks (RNNs) to accurately process sequential inputs with long-time dependencies is very challenging on account of the exploding and vanishing gradient problem. To overcome this, we propose a novel RNN architecture which is based on a structure preserving discretization of a Hamiltonian system of second-order ordinary differential equations that models networks of oscillators. The resulting RNN is fast, invertible (in time), memory efficient and we derive rigorous bounds on the hidden state gradients to prove the mitigation of the exploding and vanishing gradient problem. A suite of experiments are presented to demonstrate that the proposed RNN provides state of the art performance on a variety of learning tasks with (very) long-time dependencies.

Key facts. Cited by 56 papers in the map (data through 7 Oct 2026). Builds on 30 of its 50 references that are in the map. First posted on arXiv 9 Mar 2021.

Builds on

30 of 50 references.

References · 30

  1. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  2. 2018Neural Ordinary Differential Equationscited by 3,733
  3. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  4. 2019Normalizing Flows for Probabilistic Modeling and Inferencecited by 1,315
  5. 2019Hamiltonian Neural Networkscited by 702
  6. 2018The UEA multivariate time series classification archive, 2018cited by 330
  7. 2015Unitary Evolution Recurrent Neural Networkscited by 291
  8. 2017Gate-Variants of Gated Recurrent Unit (GRU) Neural Networkscited by 239
  9. 2015A Theoretically Grounded Application of Dropout in Recurrent Neural Networkscited by 161
  10. 2019Symplectic Recurrent Neural Networkscited by 148
  11. 2018Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNNcited by 142
  12. 2019Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Groupcited by 133
18 more
  1. 2015A Simple Way to Initialize Recurrent Networks of Rectified Linear Unitscited by 130
  2. 2019AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networkscited by 112
  3. 2016Full-Capacity Unitary Recurrent Neural Networkscited by 90
  4. 2020Lipschitz Recurrent Neural Networkscited by 88
  5. 2017Dilated Recurrent Neural Networkscited by 83
  6. 2017Orthogonal Recurrent Neural Networks with Scaled Cayley Transformcited by 82
  7. 2020Coupled Oscillatory Recurrent Neural Network (coRNN): An accurate and (gradient) stable architecture for learning long time dependenciescited by 81
  8. 2019Trivializations for Gradient-Based Optimization on Manifoldscited by 78
  9. 2017Simple Recurrent Units for Highly Parallelizable Recurrencecited by 60
  10. 2017Skip RNN: Learning to Skip State Updates in Recurrent Neural Networkscited by 49
  11. 2016Recurrent Orthogonal Networks and Long-Memory Taskscited by 38
  12. 2019FastGRNN: A Fast, Accurate, Stable and Tiny Kilobyte Sized Gated Recurrent Neural Networkcited by 38
  13. 2019Non-normal Recurrent Neural Network (nnRNN): learning long time dependencies while improving expressivity with transient dynamicscited by 28
  14. 2018Reversible Recurrent Neural Networkscited by 22
  15. 2019Deep Independently Recurrent Neural Network (IndRNN)cited by 14
  16. 2016A recurrent neural network without chaoscited by 13
  17. 2016Optimizing Performance of Recurrent Neural Networks on GPUscited by 12
  18. 2020Monash University, UEA, UCR Time Series Extrinsic Regression Archivecited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 10; 2023: 11; 2024: 14; 2025: 11; 2026: 4.

56 papers.

Sort:

By area:

  1. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  2. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  3. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 607
  4. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 418
  5. 2023On Over-Squashing in Message Passing Neural Networks: The Impact of Width, Depth, and Topologycited by 163
  6. 2022Liquid Structural State-Space Modelscited by 134
  7. 2022Graph-Coupled Oscillator Networkscited by 123
  8. 2021FlexConv: Continuous Kernel Convolutions with Differentiable Kernel Sizescited by 66
  9. 2021On the difficulty of learning chaotic dynamics with RNNscited by 64
  10. 2022Gradient Gating for Deep Multi-Rate Learning on Graphscited by 49
  11. 2023Parallelizing non-linear sequential models over the sequence lengthcited by 45
  12. 2021Long Expressive Memory for Sequence Modelingcited by 41
38 more
  1. 2024Oscillatory State-Space Modelscited by 35
  2. 2023Robustifying State-space Models for Long Sequences via Approximate Diagonalizationcited by 18
  3. 2023Neural Oscillators are Universalcited by 16
  4. 2023Traveling Waves Encode the Recent Past and Enhance Sequence Learningcited by 16
  5. 2024Rough Transformers: Lightweight and Continuous Time Series Modelling through Signature Patchingcited by 16
  6. 2024Tuning Frequency Bias of State Space Modelscited by 16
  7. 2022Neural Differential Equations for Learning to Program Neural Nets Through Continuous Learning Rulescited by 15
  8. 2021Stateful ODE-Nets using Basis Function Expansionscited by 14
  9. 2022Dynamical systems' based neural networkscited by 10
  10. 2023Neural oscillators for generalization of physics-informed machine learningcited by 9
  11. 2024Input-to-State Stable Coupled Oscillator Networks for Closed-form Model-based Control in Latent Spacecited by 7
  12. 2025Block-Biased Mamba for Long-Range Sequence Processingcited by 7
  13. 2025Learning to Dissipate Energy in Oscillatory State-Space Modelscited by 7
  14. 2022Approximation of nearly-periodic symplectic maps via structure-preserving neural networkscited by 6
  15. 2023Persistent learning signals and working memory without continuous attractorscited by 6
  16. 2025Learning long range dependencies through time reversal symmetry breakingcited by 6
  17. 2025Over-squashing in Spatiotemporal Graph Neural Networkscited by 6
  18. 2024Deep learning in motion deblurring: current status, benchmarks and future prospectscited by 5
  19. 2022Sequence Learning Using Equilibrium Propagationcited by 4
  20. 2024Unconditional stability of a recurrent neural circuit implementing divisive normalizationcited by 4
  21. 2023Learning to Decouple Complex Systemscited by 3
  22. 2025A Deep State Space Model for Rainfall-Runoff Simulationscited by 3
  23. 2025Stuart-Landau Oscillatory Graph Neural Networkcited by 3
  24. 2023Trainability, Expressivity and Interpretability in Gated Neural ODEscited by 2
  25. 2025Myosotis: structured computation for attention like layercited by 2
  26. 2025Hankel Singular Value Regularization for Highly Compressible State Space Modelscited by 2
  27. 2023Neural oscillators for magnetic hysteresis modelingcited by 1
  28. 2023Delving Deeper Into Astromorphic Transformerscited by 1
  29. 2024Input Convex Lipschitz Recurrent Neural Networks for Robust and Efficient Process Modeling and Optimizationcited by 1
  30. 2024Quantized Approximately Orthogonal Recurrent Neural Networkscited by 1
  31. 2024Towards Foundation Model for Chemical Reactor Modeling: Meta-Learning with Physics-Informed Adaptationcited by 1
  32. 2024Birdie: Advancing State Space Models with Reward-Driven Objectives and Curriculacited by 1
  33. 2025Upper Approximation Bounds for Neural Oscillatorscited by 1
  34. 2026Metriplector: From Field Theory to Neural Architecturecited by 1
  35. 2022Fast Saturating Gate for Learning Long Time Scales with Recurrent Neural Networkscited by 0
  36. 2024Topological Neural Networks: Mitigating the Bottlenecks of Graph Neural Networks via Higher-Order Interactionscited by 0
  37. 2024Efficient State Space Model via Fast Tensor Convolution and Block Diagonalizationcited by 0
  38. 2024Neural Controlled Differential Equations with Quantum Hidden Evolutionscited by 0

50 most cited of 56 All 56 (JSON)