# Hierarchical Deep Learning of Multiscale Differential Equation Time-Steppers

arXiv: https://arxiv.org/abs/2008.09768 · HTML: https://mlsanity.com/paper/2008.09768/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Aug 2020.

## Built on by (most cited first, 30 of 37)

- [Learning Dissipative Dynamics in Chaotic Systems](https://mlsanity.com/paper/2106.06898/) (2021)
- [Physics and Equality Constrained Artificial Neural Networks: Application to Forward and Inverse Problems with Multi-fidelity Data Fusion](https://mlsanity.com/paper/2109.14860/) (2021)
- [Structure-preserving Sparse Identification of Nonlinear Dynamics for Data-driven Modeling](https://mlsanity.com/paper/2109.05364/) (2021)
- [Reduced Order Modeling with Shallow Recurrent Decoder Networks](https://mlsanity.com/paper/2502.10930/) (2025)
- [Renormalizing Diffusion Models](https://mlsanity.com/paper/2308.12355/) (2023)
- [When are dynamical systems learned from time series data statistically accurate?](https://mlsanity.com/paper/2411.06311/) (2024)
- [Multi-scale Time-stepping of Partial Differential Equations with Transformers](https://mlsanity.com/paper/2311.02225/) (2023)
- [MORPH: PDE Foundation Models with Arbitrary Data Modality](https://mlsanity.com/paper/2509.21670/) (2025)
- [Feature-adjacent multi-fidelity physics-informed machine learning for partial differential equations](https://mlsanity.com/paper/2303.11577/) (2023)
- [Adaptive physics-informed neural operator for coarse-grained non-equilibrium flows](https://mlsanity.com/paper/2210.15799/) (2022)
- [Leveraging Jumpy Models for Planning and Fast Learning in Robotic Domains](https://mlsanity.com/paper/2302.12617/) (2023)
- [Learning Neural Differential Algebraic Equations via Operator Splitting](https://mlsanity.com/paper/2403.12938/) (2024)
- [Deep Surrogate for Direct Time Fluid Dynamics](https://mlsanity.com/paper/2112.10296/) (2021)
- [Using Probabilistic Machine Learning to Better Model Temporal Patterns in Parameterizations: a case study with the Lorenz 96 model](https://mlsanity.com/paper/2203.14814/) (2022)
- [Defining error accumulation in ML atmospheric simulators](https://mlsanity.com/paper/2405.14714/) (2024)
- [Adaptive learning of effective dynamics: Adaptive real-time, online modeling for complex systems](https://mlsanity.com/paper/2304.01732/) (2023)
- [Latent Dynamics Networks (LDNets): learning the intrinsic dynamics of spatio-temporal processes](https://mlsanity.com/paper/2305.00094/) (2023)
- [Hierarchical deep learning-based adaptive time-stepping scheme for multiscale simulations](https://mlsanity.com/paper/2311.05961/) (2023)
- [A Physics-informed machine learning model for time-dependent wave runup prediction](https://mlsanity.com/paper/2401.08684/) (2024)
- [A Two-Phase Deep Learning Framework for Adaptive Time-Stepping in High-Speed Flow Modeling](https://mlsanity.com/paper/2506.07969/) (2025)
- [Efficient time stepping for numerical integration using reinforcement learning](https://mlsanity.com/paper/2104.03562/) (2021)
- [Taylorformer: Probabilistic Modelling for Random Processes including Time Series](https://mlsanity.com/paper/2305.19141/) (2023)
- [Advancing Spatio-temporal Storm Surge Prediction with Hierarchical Deep Neural Networks](https://mlsanity.com/paper/2410.12823/) (2024)
- [A Physics-Informed Fourier-Wavelet Transformer for Multiscale Computational Fluid Dynamics Surrogate Modeling](https://mlsanity.com/paper/2606.24696/) (2026)
- [Accelerating Dynamical System Simulations with Contracting and Physics-Projected Neural-Newton Solvers](https://mlsanity.com/paper/2106.02543/) (2021)
- [Algorithmically Designed Artificial Neural Networks (ADANNs): Higher order deep operator learning for parametric partial differential equations](https://mlsanity.com/paper/2302.03286/) (2023)
- [InvKA: Gait Recognition via Invertible Koopman Autoencoder](https://mlsanity.com/paper/2309.14764/) (2023)
- [Multi-Hierarchical Surrogate Learning for Structural Dynamical Crash Simulations Using Graph Convolutional Neural Networks](https://mlsanity.com/paper/2402.09234/) (2024)
- [Regularization of ML models for Earth systems by using longer model timesteps](https://mlsanity.com/paper/2503.18023/) (2025)
- [Double projection for reconstructing dynamical systems: between stochastic and deterministic regimes](https://mlsanity.com/paper/2510.01089/) (2025)

## Abstract (arXiv)

Nonlinear differential equations rarely admit closed-form solutions, thus requiring numerical time-stepping algorithms to approximate solutions. Further, many systems characterized by multiscale physics exhibit dynamics over a vast range of timescales, making numerical integration computationally expensive due to numerical stiffness. In this work, we develop a hierarchy of deep neural network time-steppers to approximate the flow map of the dynamical system over a disparate range of time-scales. The resulting model is purely data-driven and leverages features of the multiscale dynamics, enabling numerical integration and forecasting that is both accurate and highly efficient. Moreover, similar ideas can be used to couple neural network-based models with classical numerical time-steppers. Our multiscale hierarchical time-stepping scheme provides important advantages over current time-stepping algorithms, including (i) circumventing numerical stiffness due to disparate time-scales, (ii) improved accuracy in comparison with leading neural-network architectures, (iii) efficiency in long-time simulation/forecasting due to explicit training of slow time-scale dynamics, and (iv) a flexible framework that is parallelizable and may be integrated with standard numerical time-stepping algorithms. The method is demonstrated on a wide range of nonlinear dynamical systems, including the Van der Pol oscillator, the Lorenz system, the Kuramoto-Sivashinsky equation, and fluid flow pass a cylinder; audio and video signals are also explored. On the sequence generation examples, we benchmark our algorithm against state-of-the-art methods, such as LSTM, reservoir computing, and clockwork RNN. Despite the structural simplicity of our method, it outperforms competing methods on numerical integration.