# Randomized Sparse Neural Galerkin Schemes for Solving Evolution Equations with Deep Networks

arXiv: https://arxiv.org/abs/2310.04867 · HTML: https://mlsanity.com/paper/2310.04867/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 54 references that are in the map.
- First posted on arXiv 7 Oct 2023.

## Builds on (most cited first, 27 of 27)

- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://mlsanity.com/paper/2106.10199/) (2021)
- [DGM: A deep learning algorithm for solving partial differential equations](https://mlsanity.com/paper/1708.07469/) (2017)
- [Characterizing possible failure modes in physics-informed neural networks](https://mlsanity.com/paper/2109.01050/) (2021)
- [Solving high-dimensional partial differential equations using deep learning](https://mlsanity.com/paper/1707.02568/) (2017)
- [Machine learning accelerated computational fluid dynamics](https://mlsanity.com/paper/2102.01010/) (2021)
- [Deep Potential Molecular Dynamics: a scalable model with the accuracy of quantum mechanics](https://mlsanity.com/paper/1707.09571/) (2017)
- [Training Neural Networks with Fixed Sparse Masks](https://mlsanity.com/paper/2111.09839/) (2021)
- [Respecting causality is all you need for training physics-informed neural networks](https://mlsanity.com/paper/2203.07404/) (2022)
- [Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Models](https://mlsanity.com/paper/1909.11299/) (2019)
- [Exact imposition of boundary conditions with distance functions in physics-informed deep neural networks](https://mlsanity.com/paper/2104.08426/) (2021)
- [A unified deep artificial neural network approach to partial differential equations in complex geometries](https://mlsanity.com/paper/1711.06464/) (2017)
- [Reducing Overfitting in Deep Networks by Decorrelating Representations](https://mlsanity.com/paper/1511.06068/) (2015)
- [A Method for Representing Periodic Functions and Enforcing Exactly Periodic Boundary Conditions with Deep Neural Networks](https://mlsanity.com/paper/2007.07442/) (2020)
- [POD-DL-ROM: enhancing deep learning-based reduced order models for nonlinear parametrized PDEs by proper orthogonal decomposition](https://mlsanity.com/paper/2101.11845/) (2021)
- [Algorithms for Solving High Dimensional PDEs: From Nonlinear Monte Carlo to Machine Learning](https://mlsanity.com/paper/2008.13333/) (2020)
- [Rational neural networks](https://mlsanity.com/paper/2004.01902/) (2020)
- [Can Physics-Informed Neural Networks beat the Finite Element Method?](https://mlsanity.com/paper/2302.04107/) (2023)
- [Neural Galerkin Schemes with Active Learning for High-Dimensional Evolution Equations](https://mlsanity.com/paper/2203.01360/) (2022)
- [Evolutional Deep Neural Network](https://mlsanity.com/paper/2103.09959/) (2021)
- [Solving for high dimensional committor functions using artificial neural networks](https://mlsanity.com/paper/1802.10275/) (2018)
- [Computing Committor Functions for the Study of Rare Events Using Deep Learning](https://mlsanity.com/paper/1906.06285/) (2019)
- [Regularization techniques for fine-tuning in neural machine translation](https://mlsanity.com/paper/1707.09920/) (2017)
- [An Extensible Benchmark Suite for Learning to Simulate Physical Systems](https://mlsanity.com/paper/2108.07799/) (2021)
- [A Stable and Scalable Method for Solving Initial Value PDEs with Neural Networks](https://mlsanity.com/paper/2304.14994/) (2023)
- [Model reduction for the material point method via an implicit neural representation of the deformation map](https://mlsanity.com/paper/2109.12390/) (2021)

## Built on by (most cited first, 18 of 18)

- [CoLoRA: Continuous low-rank adaptation for reduced implicit neural modeling of parameterized partial differential equations](https://mlsanity.com/paper/2402.14646/) (2024)
- [Sequential-in-time training of nonlinear parametrizations for solving time-dependent partial differential equations](https://mlsanity.com/paper/2404.01145/) (2024)
- [Wave-RVFL: A Randomized Neural Network Based on Wave Loss Function](https://mlsanity.com/paper/2408.02824/) (2024)
- [Operator learning for hyperbolic partial differential equations](https://mlsanity.com/paper/2312.17489/) (2023)
- [TENG: Time-Evolving Natural Gradient for Solving PDEs With Deep Neural Nets Toward Machine Precision](https://mlsanity.com/paper/2404.10771/) (2024)
- [Nonlinear embeddings for conserving Hamiltonians and other quantities with Neural Galerkin schemes](https://mlsanity.com/paper/2310.07485/) (2023)
- [Randomized based restricted kernel machine for hyperspectral image classification](https://mlsanity.com/paper/2503.05837/) (2025)
- [A hybrid FEM-PINN method for time-dependent partial differential equations](https://mlsanity.com/paper/2409.02810/) (2024)
- [BWLer: Barycentric Weight Layer Elucidates a Precision-Conditioning Tradeoff for PINNs](https://mlsanity.com/paper/2506.23024/) (2025)
- [Low-Rank Evolutionary Deep Neural Networks via Adaptive Tangent-Space Reduction](https://mlsanity.com/paper/2509.16395/) (2025)
- [Filtered Neural Galerkin model reduction schemes for efficient propagation of initial condition uncertainties in digital twins](https://mlsanity.com/paper/2511.00670/) (2025)
- [Nonlinear model reduction for transport-dominated problems](https://mlsanity.com/paper/2602.01397/) (2026)
- [A Dirac-Frenkel-Onsager principle: Instantaneous residual minimization with gauge momentum for nonlinear parametrizations of PDE solutions](https://mlsanity.com/paper/2605.00284/) (2026)
- [Solving PDEs With Deep Neural Nets under General Boundary Conditions](https://mlsanity.com/paper/2512.15771/) (2025)
- [MAD-NG: Meta-Auto-Decoder Neural Galerkin Method for Solving Parametric Partial Differential Equations](https://mlsanity.com/paper/2512.21633/) (2025)
- [TENG-BC: Unified Time-Evolving Natural Gradient for Neural PDE Solvers with General Boundary Conditions](https://mlsanity.com/paper/2603.00397/) (2026)
- [Stochastic-Dimension Frozen Sampled Neural Network for High-Dimensional Gross-Pitaevskii Equations on Unbounded Domains](https://mlsanity.com/paper/2604.09361/) (2026)
- [Dirac-Frenkel dynamics with inertia for nonlinearly parametrized solutions of evolution problems](https://mlsanity.com/paper/2606.24769/) (2026)

## Abstract (arXiv)

Training neural networks sequentially in time to approximate solution fields of time-dependent partial differential equations can be beneficial for preserving causality and other physics properties; however, the sequential-in-time training is numerically challenging because training errors quickly accumulate and amplify over time. This work introduces Neural Galerkin schemes that update randomized sparse subsets of network parameters at each time step. The randomization avoids overfitting locally in time and so helps prevent the error from accumulating quickly over the sequential-in-time training, which is motivated by dropout that addresses a similar issue of overfitting due to neuron co-adaptation. The sparsity of the update reduces the computational costs of training without losing expressiveness because many of the network parameters are redundant locally at each time step. In numerical experiments with a wide range of evolution equations, the proposed scheme with randomized sparse updates is up to two orders of magnitude more accurate at a fixed computational budget and up to two orders of magnitude faster at a fixed accuracy than schemes with dense updates.