# Divide And Conquer: Learning Chaotic Dynamical Systems With Multistep Penalty Neural Ordinary Differential Equations

arXiv: https://arxiv.org/abs/2407.00568 · HTML: https://mlsanity.com/paper/2407.00568/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 99 references that are in the map.
- First posted on arXiv 30 Jun 2024.

## Builds on (most cited first, 30 of 30)

- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Averaging Weights Leads to Wider Optima and Better Generalization](https://mlsanity.com/paper/1803.05407/) (2018)
- [Neural Controlled Differential Equations for Irregular Time Series](https://mlsanity.com/paper/2005.08926/) (2020)
- [Message Passing Neural PDE Solvers](https://mlsanity.com/paper/2202.03376/) (2022)
- [Unitary Evolution Recurrent Neural Networks](https://mlsanity.com/paper/1511.06464/) (2015)
- [A Critical Review of Recurrent Neural Networks for Sequence Learning](https://mlsanity.com/paper/1506.00019/) (2015)
- [Neural Stochastic Differential Equations: Deep Latent Gaussian Models in the Diffusion Limit](https://mlsanity.com/paper/1905.09883/) (2019)
- [Forecasting Global Weather with Graph Neural Networks](https://mlsanity.com/paper/2202.07575/) (2022)
- [Graph Neural Ordinary Differential Equations](https://mlsanity.com/paper/1911.07532/) (2019)
- [Dissecting Neural ODEs](https://mlsanity.com/paper/2002.08071/) (2020)
- [AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks](https://mlsanity.com/paper/1902.09689/) (2019)
- [Neural SDE: Stabilizing Neural ODE Networks with Stochastic Noise](https://mlsanity.com/paper/1906.02355/) (2019)
- [Which Neural Net Architectures Give Rise To Exploding and Vanishing Gradients?](https://mlsanity.com/paper/1801.03744/) (2018)
- [Learned Coarse Models for Efficient Turbulence Simulation](https://mlsanity.com/paper/2112.15275/) (2021)
- [On Second Order Behaviour in Augmented Neural ODEs](https://mlsanity.com/paper/2006.07220/) (2020)
- [On the difficulty of learning chaotic dynamics with RNNs](https://mlsanity.com/paper/2110.07238/) (2021)
- [STEER: Simple Temporal Regularization For Neural ODEs](https://mlsanity.com/paper/2006.10711/) (2020)
- [ACE: A fast, skillful learned global atmospheric model for climate prediction](https://mlsanity.com/paper/2310.02074/) (2023)
- [MALI: A memory efficient and reverse accurate integrator for Neural ODEs](https://mlsanity.com/paper/2102.04668/) (2021)
- [DySLIM: Dynamics Stable Learning by Invariant Measure for Chaotic Systems](https://mlsanity.com/paper/2402.04467/) (2024)
- [LUCIE: A Lightweight Uncoupled ClImate Emulator with long-term stability and physical consistency for O(1000)-member ensembles](https://mlsanity.com/paper/2405.16297/) (2024)
- [ChemNODE: A Neural Ordinary Differential Equations Approach for Chemical Kinetics Solvers](https://mlsanity.com/paper/2101.04749/) (2020)
- [Latent Neural ODEs with Sparse Bayesian Multiple Shooting](https://mlsanity.com/paper/2210.03466/) (2022)
- [The exploding gradient problem demystified - definition, prevalence, impact, origin, tradeoffs, and solutions](https://mlsanity.com/paper/1712.05577/) (2017)
- [A Review of Designs and Applications of Echo State Networks](https://mlsanity.com/paper/2012.02974/) (2020)
- [Universal Battery Performance and Degradation Model for Electric Aircraft](https://mlsanity.com/paper/2008.01527/) (2020)
- [Learning Dynamical Systems from Noisy Sensor Measurements using Multiple Shooting](https://mlsanity.com/paper/2106.11712/) (2021)
- [Regularization and Reparameterization Avoid Vanishing Gradients in Sigmoid-Type Networks](https://mlsanity.com/paper/2106.02260/) (2021)
- [Effective Latent Differential Equation Models via Attention and Multiple Shooting](https://mlsanity.com/paper/2307.05735/) (2023)
- [Identifying nonlinear dynamical systems with multiple time scales and long-range dependencies](https://mlsanity.com/paper/1910.03471/) (2019)

## Built on by (most cited first, 1 of 1)

- [Physics-constrained coupled neural differential equations for one dimensional blood flow modeling](https://mlsanity.com/paper/2411.05631/) (2024)

## Abstract (arXiv)

Forecasting high-dimensional dynamical systems is a fundamental challenge in various fields, such as geosciences and engineering. Neural Ordinary Differential Equations (NODEs), which combine the power of neural networks and numerical solvers, have emerged as a promising algorithm for forecasting complex nonlinear dynamical systems. However, classical techniques used for NODE training are ineffective for learning chaotic dynamical systems. In this work, we propose a novel NODE-training approach that allows for robust learning of chaotic dynamical systems. Our method addresses the challenges of non-convexity and exploding gradients associated with underlying chaotic dynamics. Training data trajectories from such systems are split into multiple, non-overlapping time windows. In addition to the deviation from the training data, the optimization loss term further penalizes the discontinuities of the predicted trajectory between the time windows. The window size is selected based on the fastest Lyapunov time scale of the system. Multi-step penalty(MP) method is first demonstrated on Lorenz equation, to illustrate how it improves the loss landscape and thereby accelerates the optimization convergence. MP method can optimize chaotic systems in a manner similar to least-squares shadowing with significantly lower computational costs. Our proposed algorithm, denoted the Multistep Penalty NODE, is applied to chaotic systems such as the Kuramoto-Sivashinsky equation, the two-dimensional Kolmogorov flow, and ERA5 reanalysis data for the atmosphere. It is observed that MP-NODE provide viable performance for such chaotic systems, not only for short-term trajectory predictions but also for invariant statistics that are hallmarks of the chaotic nature of these dynamics.