# Numerical Approximation of Partial Differential Equations by a Variable Projection Method with Artificial Neural Networks arXiv: https://arxiv.org/abs/2201.09989 ยท HTML: https://mlsanity.com/paper/2201.09989/ ## Key facts - Cited by 9 papers in the map (data through 14 Oct 2026). - Builds on 22 of its 81 references that are in the map. - First posted on arXiv 24 Jan 2022. ## Builds on (most cited first, 22 of 22) - [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014) - [DGM: A deep learning algorithm for solving partial differential equations](https://mlsanity.com/paper/1708.07469/) (2017) - [Characterizing possible failure modes in physics-informed neural networks](https://mlsanity.com/paper/2109.01050/) (2021) - [When and why PINNs fail to train: A neural tangent kernel perspective](https://mlsanity.com/paper/2007.14527/) (2020) - [DeepXDE: A deep learning library for solving differential equations](https://mlsanity.com/paper/1907.04502/) (2019) - [The Deep Ritz method: A deep learning-based numerical algorithm for solving variational problems](https://mlsanity.com/paper/1710.00211/) (2017) - [An Energy Approach to the Solution of Partial Differential Equations in Computational Mechanics via Machine Learning: Concepts, Implementation and Applications](https://mlsanity.com/paper/1908.10407/) (2019) - [A Method for Representing Periodic Functions and Enforcing Exactly Periodic Boundary Conditions with Deep Neural Networks](https://mlsanity.com/paper/2007.07442/) (2020) - [Local Extreme Learning Machines and Domain Decomposition for Solving Linear and Nonlinear Partial Differential Equations](https://mlsanity.com/paper/2012.02895/) (2020) - [Physics Informed Extreme Learning Machine (PIELM) -- A rapid method for the numerical solution of partial differential equations](https://mlsanity.com/paper/1907.03507/) (2019) - [Robust Training and Initialization of Deep Neural Networks: An Adaptive Basis Viewpoint](https://mlsanity.com/paper/1912.04862/) (2019) - [Two-Layer Neural Networks for Partial Differential Equations: Optimization and Generalization Theory](https://mlsanity.com/paper/2006.15733/) (2020) - [On Computing the Hyperparameter of Extreme Learning Machines: Algorithm and Application to Computational PDEs, and Comparison with Classical and High-Order Finite Elements](https://mlsanity.com/paper/2110.14121/) (2021) - [A Modified Batch Intrinsic Plasticity Method for Pre-training the Random Coefficients of Extreme Learning Machines](https://mlsanity.com/paper/2103.08042/) (2021) - [Train Like a (Var)Pro: Efficient Training of Neural Networks with Variable Projection](https://mlsanity.com/paper/2007.13171/) (2020) - [Solving PDEs on Unknown Manifolds with Machine Learning](https://mlsanity.com/paper/2106.06682/) (2021) - [Sparse Principal Component Analysis via Variable Projection](https://mlsanity.com/paper/1804.00341/) (2018) - [Data-driven learning of robust nonlocal physics from high-fidelity synthetic data](https://mlsanity.com/paper/2005.10076/) (2020) - [slimTrain -- A Stochastic Approximation Method for Training Separable Deep Neural Networks](https://mlsanity.com/paper/2109.14002/) (2021) - [A generalization of variable elimination for separable inverse problems beyond least squares](https://mlsanity.com/paper/1302.0441/) (2013) - [A Roadmap for Discretely Energy-Stable Schemes for Dissipative Systems Based on a Generalized Auxiliary Variable with Guaranteed Positivity](https://mlsanity.com/paper/1904.00141/) (2019) - [LAP: a Linearize and Project Method for Solving Inverse Problems with Coupled Variables](https://mlsanity.com/paper/1705.09992/) (2017) ## Built on by (most cited first, 9 of 9) - [An Extreme Learning Machine-Based Method for Computational PDEs in Higher Dimensions](https://mlsanity.com/paper/2309.07049/) (2023) - [Deep Learning in Deterministic Computational Mechanics](https://mlsanity.com/paper/2309.15421/) (2023) - [A Method for Computing Inverse Parametric PDE Problems with Random-Weight Neural Networks](https://mlsanity.com/paper/2210.04338/) (2022) - [Slow Invariant Manifolds of Singularly Perturbed Systems via Physics-Informed Machine Learning](https://mlsanity.com/paper/2309.07946/) (2023) - [Numerical Computation of Partial Differential Equations by Hidden-Layer Concatenated Extreme Learning Machine](https://mlsanity.com/paper/2204.11375/) (2022) - [Error Analysis of Physics-Informed Neural Networks for Approximating Dynamic PDEs of Second Order in Time](https://mlsanity.com/paper/2303.12245/) (2023) - [A Novel Method for Enforcing Exactly Dirichlet, Neumann and Robin Conditions on Curved Domain Boundaries for Physics Informed Machine Learning](https://mlsanity.com/paper/2603.21909/) (2026) - [Error Analysis and Numerical Algorithm for PDE Approximation with Hidden-Layer Concatenated Physics Informed Neural Networks](https://mlsanity.com/paper/2406.06350/) (2024) - [Constrained Variable Projection for Structured Problems](https://mlsanity.com/paper/2606.23939/) (2026) ## Abstract (arXiv) We present a method for solving linear and nonlinear PDEs based on the variable projection (VarPro) framework and artificial neural networks (ANN). For linear PDEs, enforcing the boundary/initial value problem on the collocation points leads to a separable nonlinear least squares problem about the network coefficients. We reformulate this problem by the VarPro approach to eliminate the linear output-layer coefficients, leading to a reduced problem about the hidden-layer coefficients only. The reduced problem is solved first by the nonlinear least squares method to determine the hidden-layer coefficients, and then the output-layer coefficients are computed by the linear least squares method. For nonlinear PDEs, enforcing the boundary/initial value problem on the collocation points leads to a nonlinear least squares problem that is not separable, which precludes the VarPro strategy for such problems. To enable the VarPro approach for nonlinear PDEs, we first linearize the problem with a Newton iteration, using a particular form of linearization. The linearized system is solved by the VarPro framework together with ANNs. Upon convergence of the Newton iteration, the network coefficients provide the representation of the solution field to the original nonlinear problem. We present ample numerical examples with linear and nonlinear PDEs to demonstrate the performance of the method herein. For smooth field solutions, the errors of the current method decrease exponentially as the number of collocation points or the number of output-layer coefficients increases. We compare the current method with the ELM method from a previous work. Under identical conditions and network configurations, the current method exhibits an accuracy significantly superior to the ELM method.