# Sequential-in-time training of nonlinear parametrizations for solving time-dependent partial differential equations

arXiv: https://arxiv.org/abs/2404.01145 · HTML: https://mlsanity.com/paper/2404.01145/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 94 references that are in the map.
- First posted on arXiv 1 Apr 2024.

## Builds on (most cited first, 17 of 17)

- [New insights and perspectives on the natural gradient method](https://mlsanity.com/paper/1412.1193/) (2014)
- [Quantum Natural Gradient](https://mlsanity.com/paper/1909.02108/) (2019)
- [ANODE: Unconditionally Accurate Memory-Efficient Gradients for Neural ODEs](https://mlsanity.com/paper/1902.10298/) (2019)
- [Neural Galerkin Schemes with Active Learning for High-Dimensional Evolution Equations](https://mlsanity.com/paper/2203.01360/) (2022)
- [Evolutional Deep Neural Network](https://mlsanity.com/paper/2103.09959/) (2021)
- [Discretize-Optimize vs. Optimize-Discretize for Time-Series Regression and Continuous Normalizing Flows](https://mlsanity.com/paper/2005.13420/) (2020)
- [Implicit Neural Spatial Representations for Time-dependent PDEs](https://mlsanity.com/paper/2210.00124/) (2022)
- [Sampling via Gradient Flows in the Space of Probability Measures](https://mlsanity.com/paper/2310.03597/) (2023)
- [Randomized Sparse Neural Galerkin Schemes for Solving Evolution Equations with Deep Networks](https://mlsanity.com/paper/2310.04867/) (2023)
- [A Stable and Scalable Method for Solving Initial Value PDEs with Neural Networks](https://mlsanity.com/paper/2304.14994/) (2023)
- [Optimal transport natural gradient for statistical manifolds with continuous sample space](https://mlsanity.com/paper/1805.08380/) (2018)
- [Particle-based Energetic Variational Inference](https://mlsanity.com/paper/2004.06443/) (2020)
- [CoLoRA: Continuous low-rank adaptation for reduced implicit neural modeling of parameterized partial differential equations](https://mlsanity.com/paper/2402.14646/) (2024)
- [Wasserstein Quantum Monte Carlo: A Novel Approach for Solving the Quantum Many-Body Schr"odinger Equation](https://mlsanity.com/paper/2307.07050/) (2023)
- [Model reduction for the material point method via an implicit neural representation of the deformation map](https://mlsanity.com/paper/2109.12390/) (2021)
- [Coupling parameter and particle dynamics for adaptive sampling in Neural Galerkin schemes](https://mlsanity.com/paper/2306.15630/) (2023)
- [Stable rank-adaptive Dynamically Orthogonal Runge-Kutta schemes](https://mlsanity.com/paper/2211.07852/) (2022)

## Built on by (most cited first, 7 of 7)

- [Solving Inverse Problems via Diffusion-Based Priors: An Approximation-Free Ensemble Sampling Approach](https://mlsanity.com/paper/2506.03979/) (2025)
- [Efficient, Multimodal, and Derivative-Free Bayesian Inference With Fisher-Rao Gradient Flows](https://mlsanity.com/paper/2406.17263/) (2024)
- [PINNs in PDE Constrained Optimal Control Problems: Direct vs Indirect Methods](https://mlsanity.com/paper/2604.04920/) (2026)
- [SPIKE: Stable Physics-Informed Kernel Evolution Method for Solving Hyperbolic Conservation Laws](https://mlsanity.com/paper/2510.18266/) (2025)
- [Filtered Neural Galerkin model reduction schemes for efficient propagation of initial condition uncertainties in digital twins](https://mlsanity.com/paper/2511.00670/) (2025)
- [A Dirac-Frenkel-Onsager principle: Instantaneous residual minimization with gauge momentum for nonlinear parametrizations of PDE solutions](https://mlsanity.com/paper/2605.00284/) (2026)
- [Dirac-Frenkel dynamics with inertia for nonlinearly parametrized solutions of evolution problems](https://mlsanity.com/paper/2606.24769/) (2026)

## Abstract (arXiv)

Sequential-in-time methods solve a sequence of training problems to fit nonlinear parametrizations such as neural networks to approximate solution trajectories of partial differential equations over time. This work shows that sequential-in-time training methods can be understood broadly as either optimize-then-discretize (OtD) or discretize-then-optimize (DtO) schemes, which are well known concepts in numerical analysis. The unifying perspective leads to novel stability and a posteriori error analysis results that provide insights into theoretical and numerical aspects that are inherent to either OtD or DtO schemes such as the tangent space collapse phenomenon, which is a form of over-fitting. Additionally, the unified perspective facilitates establishing connections between variants of sequential-in-time training methods, which is demonstrated by identifying natural gradient descent methods on energy functionals as OtD schemes applied to the corresponding gradient flows.