mlsanityData to 14 Oct 2026

Continual Learning of Dynamical Systems in Recurrent Neural Networks through Recyclable Unit Gating

arXiv 2609.38356 ↗First posted 29 Sep 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Dynamical Systems Reconstruction (DSR) aims to infer models from observed time series that reproduce a system's qualitative long-term behavior. Continual DSR (cDSR) requires learning new systems while preserving previously learned dynamics, yet even small parameter updates in recurrent models can qualitatively alter their behavior over long autonomous rollouts. We benchmark established continual learning (CL) methods spanning parameter regularization, replay, and parameter isolation on the fully trainable and interpretable Almost-Linear RNN (AL-RNN). Parameter isolation preserves earlier dynamics most effectively, but excessive task-specific allocations can rapidly exhaust a fixed-size network. We therefore introduce Continually-Recyclable Unit-Gating (CRUG), which conserves capacity through compact allocation and forward transfer. Differentiable gates trained with an L0-based penalty select task-specific units, while unused units are recycled for subsequent tasks. Directed connections allow later tasks to reuse earlier representations without affecting the dynamics of previously committed units. CRUG achieves the strongest reconstruction--capacity trade-off among the tested methods with zero forgetting and reliably learns a heterogeneous sequence of nonlinear and chaotic systems. Furthermore, we show that forward transfer is more pronounced and useful when tasks share similar underlying dynamics. Lastly, we demonstrate that CRUG's advantages extend beyond autonomous cDSR to sequential cognitive tasks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 29 of its 50 references that are in the map. First posted on arXiv 29 Sep 2026.

Builds on

29 of 50 references.

References · 29

  1. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  2. 2016Progressive Neural Networkscited by 1,361
  3. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,330
  4. 2017Continual Learning with Deep Generative Replaycited by 1,158
  5. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 898
  6. 2017Lifelong Learning with Dynamically Expandable Networkscited by 694
  7. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 681
  8. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 583
  9. 2018Progress & Compress: A scalable framework for continual learningcited by 520
  10. 2020Conditional Channel Gated Networks for Task-Aware Continual Learningcited by 115
  11. 2018Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilizationcited by 104
  12. 2019Continual Learning via Neural Pruningcited by 101
17 more
  1. 2021Chaos as an interpretable benchmark for forecasting and data-driven modellingcited by 88
  2. 2021On the difficulty of learning chaotic dynamics with RNNscited by 67
  3. 2023Generalized Teacher Forcing for Learning Chaotic Dynamicscited by 58
  4. 2017On Quadratic Penalties in Elastic Weight Consolidationcited by 55
  5. 2019Identifying nonlinear dynamical systems via generative recurrent neural networks with applications to fMRIcited by 41
  6. 2022Tractable Dendritic RNNs for Reconstructing Nonlinear Dynamical Systemscited by 36
  7. 2021Continual Learning for Recurrent Neural Networks: an Empirical Evaluationcited by 34
  8. 2024Almost-Linear RNNs Yield Highly Interpretable Symbolic Codes in Dynamical Systems Reconstructioncited by 19
  9. 2023Bifurcations and loss jumps in RNN trainingcited by 17
  10. 2024Learning Interpretable Hierarchical Dynamical Systems Models from Time Series Datacited by 15
  11. 2024Optimal Recurrent Network Topologies for Dynamical Systems Reconstructioncited by 8
  12. 2022Continual Learning of Dynamical Systems with Competitive Federated Reservoir Computingcited by 6
  13. 2025Uncovering the Computational Roles of Nonlinearity in Sequence Modeling Using Almost-Linear RNNscited by 5
  14. 2026Position: A Dynamical Systems Perspective is Needed to Advance Time Series Modelingcited by 4
  15. 2023Artificial Neuronal Ensembles with Learned Context Dependent Gatingcited by 3
  16. 2022Continual Learning of Multi-modal Dynamics with External Memorycited by 1
  17. 2026The Dynamic-Probabilistic Consistency Gap in Chaotic Surrogate Modelingcited by 1

Built on by

Not cited yet.