mlsanityData to 14 Oct 2026

Parallel-in-Time Training of Recurrent Neural Networks for Dynamical Systems Reconstruction

arXiv 2605.12683 ↗First posted 12 May 2026cs.LG cs.AI cs.DCcited by 0

Not on the map. Why?

Abstract (arXiv)

Reconstructing nonlinear dynamical systems (DS) from data (DSR) is a fundamental challenge in science and engineering, but it inherently relies on sequential models. Recent breakthroughs for sequential models have produced algorithms that parallelize computation along sequence length T, achieving logarithmic time complexity, O(log T). Since sequence lengths have been practically limited due to the linear runtime complexity O(T) of classical backpropagation through time, this opens new avenues for DSR. This paper studies two prominent classes of parallel-in-time algorithms for this task, both of which leverage parallel associative scans as their core computational primitive. The first class comprises models with linear yet non-autonomous dynamics and a nonlinear readout, such as modern State Space Models (SSMs), while the second consists of general nonlinear models which can be parallelized using the DEER framework. We find that the linear training-time recurrence of the first class of models imposes limitations that often hinder learning of accurate nonlinear dynamics. To address this, we augment DEER with Generalized Teacher Forcing (GTF), a novel variant within the more general nonlinear framework that ensures stable and effective learning of nonlinear dynamics across arbitrary sequence lengths. Using GTF-DEER, we investigate the benefits of training on extremely long sequences (T>104) for DSR. Our results show that access to such long trajectories significantly improves DSR if the data features long time scales. This work establishes GTF-DEER as a robust tool for data-driven discovery and underscores the largely untapped potential of long-sequence learning in modeling complex DS.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 45 of its 81 references that are in the map. First posted on arXiv 12 May 2026.

Builds on

45 of 81 references.

References · 45

  1. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  2. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  3. 2018Neural Ordinary Differential Equationscited by 3,779
  4. 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,908
  5. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  6. 2019DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operatorscited by 2,018
  7. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
  8. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 803
  9. 2022Simplified State Space Layers for Sequence Modelingcited by 788
  10. 2015Sequence Level Training with Recurrent Neural Networkscited by 576
  11. 2021Physics-Informed Neural Operator for Learning Partial Differential Equationscited by 527
  12. 2017Deep learning for universal linear embeddings of nonlinear dynamicscited by 494
33 more
  1. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 408
  2. 2021Modern Koopman Theory for Dynamical Systemscited by 276
  3. 2021Next Generation Reservoir Computingcited by 160
  4. 2018Data-Driven Forecasting of High-Dimensional Chaotic Systems with Long Short-Term Memory Networkscited by 155
  5. 2017Linearly-Recurrent Autoencoder Networks for Learning Dynamicscited by 145
  6. 2019Backpropagation Algorithms and Reservoir Computing in Recurrent Neural Networks for the Forecasting of Complex Spatiotemporal Dynamicscited by 134
  7. 2017Parallelizing Linear Recurrent Neural Nets Over Sequence Lengthcited by 127
  8. 2020Forecasting Sequential Data using Consistent Koopman Autoencoderscited by 115
  9. 2021On the difficulty of learning chaotic dynamics with RNNscited by 67
  10. 2023Generalized Teacher Forcing for Learning Chaotic Dynamicscited by 58
  11. 2023Parallelizing non-linear sequential models over the sequence lengthcited by 45
  12. 2019Identifying nonlinear dynamical systems via generative recurrent neural networks with applications to fMRIcited by 41
  13. 2021Long Expressive Memory for Sequence Modelingcited by 41
  14. 2024Towards Scalable and Stable Parallelization of Nonlinear RNNscited by 38
  15. 2022Koopman Neural Forecaster for Time Series with Temporal Distribution Shiftscited by 37
  16. 2022Tractable Dendritic RNNs for Reconstructing Nonlinear Dynamical Systemscited by 36
  17. 2023Universality of Linear Recurrences Followed by Non-linear Projections: Finite-Width Guarantees and Benefits of Complex Eigenvaluescited by 36
  18. 2022Using Machine Learning to Anticipate Tipping Points and Extrapolate to Post-Tipping Dynamics of Non-Stationary Dynamical Systemscited by 33
  19. 2024Out-of-Domain Generalization in Dynamical Systems Reconstructioncited by 33
  20. 2024Recurrent neural networks: vanishing and exploding gradients are not the end of the storycited by 25
  21. 2020Sparse Identification of Nonlinear Dynamical Systems via Reweighted ℓ1-regularized Least Squarescited by 21
  22. 2025True Zero-Shot Inference of Dynamical Systems Preserving Long-Term Statisticscited by 20
  23. 2024Almost-Linear RNNs Yield Highly Interpretable Symbolic Codes in Dynamical Systems Reconstructioncited by 19
  24. 2020DyNODE: Neural Ordinary Differential Equations for Dynamics Modeling in Continuous Controlcited by 18
  25. 2022Homotopy-based training of NeuralODEs for accurate dynamics discoverycited by 14
  26. 2022Integrating Multimodal Data for Joint Generative Modeling of Complex Dynamicscited by 14
  27. 2020Learn to Synchronize, Synchronize to Learncited by 11
  28. 2023Learning from Predictions: Fusing Training and Autoregressive Inference for Long-Term Spatiotemporal Forecastscited by 11
  29. 2024A scalable generative model for dynamical system reconstruction from neuroimaging datacited by 9
  30. 2025Predictability Enables Parallelization of Nonlinear State Space Modelscited by 7
  31. 2024Balanced Neural ODEs: nonlinear model order reduction and Koopman operator approximationscited by 4
  32. 2025Invariant Measures for Data-Driven Dynamical System Identification: Analysis and Applicationcited by 1
  33. 2026Tuning the burn-in phase in training recurrent neural networks improves their performancecited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025ParaRNN: Unlocking Parallel Training of Nonlinear RNNs for Large Language Models
  2. 2026Riccati State Space Models: Non-iterative Parallelization for Nonlinear Sequence Modeling
  3. 2026Time Without Timesteps: Simulating Coupled Dynamical Systems via Self-Consistency
  4. 2026Towards Infinitely Long Neural Simulations: Self-Refining Neural Surrogate Models for Dynamical Systems
  5. 2024State-Space Modeling in Long Sequence Processing: A Survey on Recurrence in the Transformer Era
  6. 2024HiPPO-Prophecy: State-Space Models can Provably Learn Dynamical Systems in Context
  7. 2025Towards Foundational Models for Dynamical System Reconstruction: Hierarchical Meta-Learning via Mixture of Experts
  8. 2025Parallelization of Non-linear State-Space Models: Scaling Up Liquid-Resistance Liquid-Capacitance Networks for Efficient Sequence Modeling