mlsanityData to 7 Oct 2026

Learning Long-Term Dependencies in Irregularly-Sampled Time Series

arXiv 2006.04418 ↗First posted 8 Jun 2020cs.LG stat.MLcited by 82

Not on the map. Why?

Abstract (arXiv)

Recurrent neural networks (RNNs) with continuous-time hidden states are a natural fit for modeling irregularly-sampled time series. These models, however, face difficulties when the input data possess long-term dependencies. We prove that similar to standard RNNs, the underlying reason for this issue is the vanishing or exploding of the gradient during training. This phenomenon is expressed by the ordinary differential equation (ODE) representation of the hidden state, regardless of the ODE solver's choice. We provide a solution by designing a new algorithm based on the long short-term memory (LSTM) that separates its memory from its time-continuous state. This way, we encode a continuous-time dynamical flow within the RNN, allowing it to respond to inputs arriving at arbitrary time-lags while ensuring a constant error propagation through the memory path. We call these RNN models ODE-LSTMs. We experimentally show that ODE-LSTMs outperform advanced RNN-based counterparts on non-uniformly sampled data with long-term dependencies. All code and data is available at https://github.com/mlech26l/ode-lstms.

Key facts. Cited by 82 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 16; 2022: 15; 2023: 17; 2024: 15; 2025: 12; 2026: 7.

82 papers.

Sort:

By area:

  1. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 607
  2. 2022Liquid Structural State-Space Modelscited by 134
  3. 2021Modeling Irregular Time Series with Continuous Recurrent Unitscited by 101
  4. 2021Neural Flows: Efficient Alternative to Neural ODEscited by 99
  5. 2021Closed-form Continuous-time Neural Modelscited by 65
  6. 2023Dataset Distillation with Convexified Implicit Gradientscited by 46
  7. 2022STDEN: Towards Physics-Guided Neural Networks for Traffic Flow Predictioncited by 45
  8. 2021Heavy Ball Neural Ordinary Differential Equationscited by 42
  9. 2024Stable Neural Stochastic Differential Equations in Analyzing Irregular Time Series Datacited by 35
  10. 2022Efficient recurrent architectures through activity sparsity and sparse back-propagation through timecited by 26
  11. 2021NRTSI: Non-Recurrent Time Series Imputationcited by 25
  12. 2024AirPhyNet: Harnessing Physics-Guided Neural Networks for Air Quality Predictioncited by 25
38 more
  1. 2024Air Quality Prediction with Physics-Guided Dual Neural ODEs in Open Systemscited by 25
  2. 2021Causal Navigation by Continuous-time Neural Networkscited by 23
  3. 2021Latent Imagination Facilitates Zero-Shot Transfer in Autonomous Racingcited by 22
  4. 2024Trajectory Flow Matching with Applications to Clinical Time Series Modelingcited by 21
  5. 2021Adversarial Training is Not Ready for Robot Learningcited by 20
  6. 2025Comprehensive Review of Neural Differential Equations for Time Series Analysiscited by 20
  7. 2023DuETT: Dual Event Time Transformer for Electronic Health Recordscited by 17
  8. 2024S7: Selective and Simplified State Space Layers for Sequence Modelingcited by 17
  9. 2024Rough Transformers: Lightweight and Continuous Time Series Modelling through Signature Patchingcited by 16
  10. 2021Sparse Flows: Pruning Continuous-depth Modelscited by 13
  11. 2021GoTube: Scalable Stochastic Verification of Continuous-Depth Modelscited by 12
  12. 2023Reduced-Order Autodifferentiable Ensemble Kalman Filterscited by 12
  13. 2024ODE Discovery for Longitudinal Heterogeneous Treatment Effects Inferencecited by 12
  14. 2021BarrierNet: A Safety-Guaranteed Layer for Neural Networkscited by 11
  15. 2023Graph-based Multi-ODE Neural Networks for Spatio-Temporal Traffic Forecastingcited by 11
  16. 2024Unveiling Delay Effects in Traffic Forecasting: A Perspective from Spatial-Temporal Delay Differential Equationscited by 11
  17. 2021How Does Momentum Benefit Deep Neural Networks Architecture Design? A Few Case Studiescited by 10
  18. 2022Differentiable Control Barrier Functions for Vision-based End-to-End Autonomous Drivingcited by 10
  19. 2022Stop&Hop: Early Classification of Irregular Time Seriescited by 10
  20. 2022On the Forward Invariance of Neural ODEscited by 10
  21. 2023Inverse Approximation Theory for Nonlinear Recurrent Neural Networkscited by 10
  22. 2023Enhancing Continuous Time Series Modelling with a Latent ODE-LSTM Approachcited by 9
  23. 2023ODE-based Recurrent Model-free Reinforcement Learning for POMDPscited by 6
  24. 2021On the Memory Mechanism of Tensor-Power Recurrent Modelscited by 5
  25. 2022Are All Vision Models Created Equal? A Study of the Open-Loop to Closed-Loop Causality Gapcited by 5
  26. 2022PyHopper -- Hyperparameter optimizationcited by 5
  27. 2024LaTiM: Longitudinal representation learning in continuous-time models to predict disease progressioncited by 5
  28. 2022Entangled Residual Mappingscited by 4
  29. 2022SeqLink: A Robust Neural-ODE Architecture for Modelling Partially Observed Time Seriescited by 4
  30. 2024DualDynamics: Synergizing Implicit and Explicit Methods for Robust Irregular Time Series Analysiscited by 4
  31. 2023Longitudinal Self-supervised Learning Using Neural Ordinary Differential Equationcited by 3
  32. 2025Neuronal Attention Circuit (NAC) for Representation Learningcited by 3
  33. 2022Learning POD of Complex Dynamics Using Heavy-ball Neural ODEscited by 2
  34. 2022Irregularly-Sampled Time Series Modeling with Spline Networkscited by 2
  35. 2023Finding Short Signals in Long Irregular Time Series with Continuous-Time Attention Policy Networkscited by 2
  36. 2023Anamnesic Neural Differential Equations with Orthogonal Polynomial Projectionscited by 2
  37. 2023Trainability, Expressivity and Interpretability in Gated Neural ODEscited by 2
  38. 2025Modeling Irregular Astronomical Time Series with Neural Stochastic Delay Differential Equationscited by 2

50 most cited of 82 All 82 (JSON)