mlsanityData to 14 Oct 2026

Long Expressive Memory for Sequence Modeling

arXiv 2110.04744 ↗First posted 10 Oct 2021cs.LG math.DS stat.MLcited by 41

Not on the map. Why?

Abstract (arXiv)

We propose a novel method called Long Expressive Memory (LEM) for learning long-term sequential dependencies. LEM is gradient-based, it can efficiently process sequential tasks with very long-term dependencies, and it is sufficiently expressive to be able to learn complicated input-output maps. To derive LEM, we consider a system of multiscale ordinary differential equations, as well as a suitable time-discretization of this system. For LEM, we derive rigorous bounds to show the mitigation of the exploding and vanishing gradients problem, a well-known challenge for gradient-based recurrent sequential learning methods. We also prove that LEM can approximate a large class of dynamical systems to high accuracy. Our empirical results, ranging from image and time-series classification through dynamical systems prediction to speech recognition and language modeling, demonstrate that LEM outperforms state-of-the-art recurrent neural networks, gated recurrent units, and long short-term memory models.

Key facts. Cited by 41 papers in the map (data through 14 Oct 2026). Builds on 28 of its 43 references that are in the map. First posted on arXiv 10 Oct 2021.

Builds on

28 of 43 references.

References · 28

  1. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  2. 2018Neural Ordinary Differential Equationscited by 3,779
  3. 2018An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modelingcited by 1,476
  4. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 752
  5. 2018The UEA multivariate time series classification archive, 2018cited by 338
  6. 2015Unitary Evolution Recurrent Neural Networkscited by 294
  7. 2019Latent ODEs for Irregularly-Sampled Time Seriescited by 183
  8. 2015A Theoretically Grounded Application of Dropout in Recurrent Neural Networkscited by 161
  9. 2020Multiscale Deep Equilibrium Modelscited by 151
  10. 2019Symplectic Recurrent Neural Networkscited by 150
  11. 2018Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNNcited by 143
  12. 2020Neural Rough Differential Equations for Long Time Seriescited by 143
16 more
  1. 2019AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networkscited by 112
  2. 2016Full-Capacity Unitary Recurrent Neural Networkscited by 94
  3. 2020Lipschitz Recurrent Neural Networkscited by 89
  4. 2017Orthogonal Recurrent Neural Networks with Scaled Cayley Transformcited by 83
  5. 2017Dilated Recurrent Neural Networkscited by 83
  6. 2020Coupled Oscillatory Recurrent Neural Network (coRNN): An accurate and (gradient) stable architecture for learning long time dependenciescited by 82
  7. 2019Trivializations for Gradient-Based Optimization on Manifoldscited by 79
  8. 2021UnICORNN: A recurrent model for learning very long time dependenciescited by 56
  9. 2018Can recurrent neural networks warp time?cited by 55
  10. 2016Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activationscited by 54
  11. 2021Noisy Recurrent Neural Networkscited by 42
  12. 2020Continuous-in-Depth Neural Networkscited by 40
  13. 2016Recurrent Orthogonal Networks and Long-Memory Taskscited by 38
  14. 2019FastGRNN: A Fast, Accurate, Stable and Tiny Kilobyte Sized Gated Recurrent Neural Networkcited by 38
  15. 2019Non-normal Recurrent Neural Network (nnRNN): learning long time dependencies while improving expressivity with transient dynamicscited by 29
  16. 2020Monash University, UEA, UCR Time Series Extrinsic Regression Archivecited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 10; 2023: 11; 2024: 8; 2025: 9; 2026: 3.

41 papers.

Sort:

By area:

  1. 2023Generalized Teacher Forcing for Learning Chaotic Dynamicscited by 58
  2. 2022Gradient Gating for Deep Multi-Rate Learning on Graphscited by 49
  3. 2023Parallelizing non-linear sequential models over the sequence lengthcited by 45
  4. 2022Koopman Neural Forecaster for Time Series with Temporal Distribution Shiftscited by 37
  5. 2022Tractable Dendritic RNNs for Reconstructing Nonlinear Dynamical Systemscited by 36
  6. 2024Out-of-Domain Generalization in Dynamical Systems Reconstructioncited by 33
  7. 2022Learning continuous models for continuous physicscited by 28
  8. 2025True Zero-Shot Inference of Dynamical Systems Preserving Long-Term Statisticscited by 20
  9. 2024Almost-Linear RNNs Yield Highly Interpretable Symbolic Codes in Dynamical Systems Reconstructioncited by 19
  10. 2024S7: Selective and Simplified State Space Layers for Sequence Modelingcited by 17
  11. 2022Neural Differential Equations for Learning to Program Neural Nets Through Continuous Learning Rulescited by 16
  12. 2023Traveling Waves Encode the Recent Past and Enhance Sequence Learningcited by 16
29 more
  1. 2024Rough Transformers: Lightweight and Continuous Time Series Modelling through Signature Patchingcited by 16
  2. 2024Hierarchical State Space Models for Continuous Sequence-to-Sequence Modelingcited by 15
  3. 2024Inferring stochastic low-rank recurrent neural networks from neural datacited by 15
  4. 2023Improving day-ahead Solar Irradiance Time Series Forecasting by Leveraging Spatio-Temporal Contextcited by 13
  5. 2022Flipped Classroom: Effective Teaching for Time Series Forecastingcited by 12
  6. 2023Multi-Scale Message Passing Neural PDE Solverscited by 12
  7. 2023Neural oscillators for generalization of physics-informed machine learningcited by 9
  8. 2022Gated Recurrent Neural Networks with Weighted Time-Delay Feedbackcited by 5
  9. 2023Delayed Memory Unit: Modelling Temporal Dependency Through Delay Gatecited by 5
  10. 2025Uncovering the Computational Roles of Nonlinearity in Sequence Modeling Using Almost-Linear RNNscited by 5
  11. 2023Neural Differential Recurrent Neural Network with Adaptive Time Stepscited by 4
  12. 2022Lyapunov-Guided Representation of Recurrent Neural Network Performancecited by 3
  13. 2024Deep Loss Convexification for Learning Iterative Modelscited by 3
  14. 2025A Deep State Space Model for Rainfall-Runoff Simulationscited by 3
  15. 2025Detecting Invariant Manifolds in ReLU-Based RNNscited by 3
  16. 2023Trainability, Expressivity and Interpretability in Gated Neural ODEscited by 2
  17. 2025Time-Scale Coupling Between States and Parameters in Recurrent Neural Networkscited by 2
  18. 2026Spontaneous symmetry breaking and Goldstone modes for deep information propagationcited by 2
  19. 2022Realization Theory Of Recurrent Neural ODEs Using Polynomial System Embeddingscited by 1
  20. 2023Improve Long-term Memory Learning Through Rescaling the Error Temporallycited by 1
  21. 2023Neural oscillators for magnetic hysteresis modelingcited by 1
  22. 2022Modeling Nonlinear Dynamics in Continuous Time with Inductive Biases on Decay Rates and/or Frequenciescited by 0
  23. 2024WaveCastNet: Rapid Wavefield Forecasting for Earthquake Early Warning via Deep Sequence to Sequence Learningcited by 0
  24. 2025Process-Informed Forecasting of Complex Thermal Dynamics in Pharmaceutical Manufacturingcited by 0
  25. 2025Unified Implementations of Recurrent Neural Networks in Multiple Deep Learning Frameworkscited by 0
  26. 2025HPC-Driven Modeling with ML-Based Surrogates for Magnon-Photon Dynamics in Hybrid Quantum Systemscited by 0
  27. 2025Quantifying Memory Use in Reinforcement Learning with Temporal Rangecited by 0
  28. 2026Parallel-in-Time Training of Recurrent Neural Networks for Dynamical Systems Reconstructioncited by 0
  29. 2026A Minimal Interpretable Architecture for Zero-Shot Reconstruction of Dynamical Systemscited by 0