# Solving and Learning Nonlinear PDEs with Gaussian Processes

arXiv: https://arxiv.org/abs/2103.12959 · HTML: https://mlsanity.com/paper/2103.12959/

## Key facts

- Cited by 129 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 98 references that are in the map.
- First posted on arXiv 24 Mar 2021.

## Builds on (most cited first, 28 of 28)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [DGM: A deep learning algorithm for solving partial differential equations](https://mlsanity.com/paper/1708.07469/) (2017)
- [When and why PINNs fail to train: A neural tangent kernel perspective](https://mlsanity.com/paper/2007.14527/) (2020)
- [The Deep Ritz method: A deep learning-based numerical algorithm for solving variational problems](https://mlsanity.com/paper/1710.00211/) (2017)
- [Solving high-dimensional partial differential equations using deep learning](https://mlsanity.com/paper/1707.02568/) (2017)
- [Deep Kernel Learning](https://mlsanity.com/paper/1511.02222/) (2015)
- [Multipole Graph Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2006.09535/) (2020)
- [PDE-Net: Learning PDEs from Data](https://mlsanity.com/paper/1710.09668/) (2017)
- [On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks](https://mlsanity.com/paper/2012.10047/) (2020)
- [Model Reduction and Neural Networks for Parametric PDEs](https://mlsanity.com/paper/2005.03180/) (2020)
- [Bayesian Deep Convolutional Encoder-Decoder Networks for Surrogate Modeling and Uncertainty Quantification](https://mlsanity.com/paper/1801.06879/) (2018)
- [The Random Feature Model for Input-Output Maps between Banach Spaces](https://mlsanity.com/paper/2005.10224/) (2020)
- [Numerical Gaussian Processes for Time-dependent and Non-linear Partial Differential Equations](https://mlsanity.com/paper/1703.10230/) (2017)
- [Probabilistic Numerics and Uncertainty in Computations](https://mlsanity.com/paper/1506.01326/) (2015)
- [A Survey of Constrained Gaussian Process Regression: Approaches and Implementation Challenges](https://mlsanity.com/paper/2006.09319/) (2020)
- [Bayesian Probabilistic Numerical Methods](https://mlsanity.com/paper/1702.03673/) (2017)
- [Kernel Flows: from learning kernels from data into the abyss](https://mlsanity.com/paper/1808.04475/) (2018)
- [Sparse Cholesky factorization by Kullback-Leibler minimization](https://mlsanity.com/paper/2004.14455/) (2020)
- [On the convergence of physics informed neural networks for linear second-order elliptic and parabolic type PDEs](https://mlsanity.com/paper/2004.01806/) (2020)
- [Compression, inversion, and approximate PCA of dense kernel matrices at near-linear computational complexity](https://mlsanity.com/paper/1706.02205/) (2017)
- [Probabilistic ODE Solvers with Runge-Kutta Means](https://mlsanity.com/paper/1406.2582/) (2014)
- [Probabilistic Numerical Methods for PDE-constrained Bayesian Inverse Problems](https://mlsanity.com/paper/1701.04006/) (2017)
- [Consistency of Empirical Bayes And Kernel Flow For Hierarchical Parameter Estimation](https://mlsanity.com/paper/2005.11375/) (2020)
- [Multigrid with rough coefficients and Multiresolution operator decomposition from Hierarchical Information Games](https://mlsanity.com/paper/1503.03467/) (2015)
- [Gamblets for opening the complexity-bottleneck of implicit schemes for hyperbolic and parabolic ODEs/PDEs with rough coefficients](https://mlsanity.com/paper/1606.07686/) (2016)
- [Kernel Mode Decomposition and programmable/interpretable regression networks](https://mlsanity.com/paper/1907.08592/) (2019)

## Built on by (most cited first, 30 of 129)

- [Long-time integration of parametric evolution equations with physics-informed DeepONets](https://mlsanity.com/paper/2106.05384/) (2021)
- [Tutorial on amortized optimization](https://mlsanity.com/paper/2202.00665/) (2022)
- [Kernel Methods are Competitive for Operator Learning](https://mlsanity.com/paper/2304.13202/) (2023)
- [Mitigating spectral bias for the multiscale operator learning](https://mlsanity.com/paper/2210.10890/) (2022)
- [A Framework for Machine Learning of Model Error in Dynamical Systems](https://mlsanity.com/paper/2107.06658/) (2021)
- [Sparse Cholesky Factorization for Solving Nonlinear PDEs via Gaussian Processes](https://mlsanity.com/paper/2304.01294/) (2023)
- [Computational Graph Completion](https://mlsanity.com/paper/2110.10323/) (2021)
- [Approximate Bayesian Neural Operators: Uncertainty Quantification for Parametric PDEs](https://mlsanity.com/paper/2208.01565/) (2022)
- [Partial Differential Equations Meet Deep Neural Networks: A Survey](https://mlsanity.com/paper/2211.05567/) (2022)
- [Error Analysis of Kernel/GP Methods for Nonlinear and Parametric PDEs](https://mlsanity.com/paper/2305.04962/) (2023)
- [AutoIP: A United Framework to Integrate Physics into Gaussian Processes](https://mlsanity.com/paper/2202.12316/) (2022)
- [Sampling via Gradient Flows in the Space of Probability Measures](https://mlsanity.com/paper/2310.03597/) (2023)
- [Parsimonious Physics-Informed Random Projection Neural Networks for Initial-Value Problems of ODEs and index-1 DAEs](https://mlsanity.com/paper/2203.05337/) (2022)
- [Recipes for when Physics Fails: Recovering Robust Learning of Physics Informed Neural Networks](https://mlsanity.com/paper/2110.13330/) (2021)
- [Learning black- and gray-box chemotactic PDEs/closures from agent based Monte Carlo simulation data](https://mlsanity.com/paper/2205.13545/) (2022)
- [A Framework Based on Symbolic Regression Coupled with eXtended Physics-Informed Neural Networks for Gray-Box Learning of Equations of Motion from Data](https://mlsanity.com/paper/2305.10706/) (2023)
- [Operator Learning with Gaussian Processes](https://mlsanity.com/paper/2409.04538/) (2024)
- [Gaussian Process Priors for Systems of Linear Partial Differential Equations with Constant Coefficients](https://mlsanity.com/paper/2212.14319/) (2022)
- [Gradient Flows for Sampling: Mean-Field Models, Gaussian Approximations and Affine Invariance](https://mlsanity.com/paper/2302.11024/) (2023)
- [Bayesian Numerical Methods for Nonlinear Partial Differential Equations](https://mlsanity.com/paper/2104.12587/) (2021)
- [Data-driven Control of Agent-based Models: an Equation/Variable-free Machine Learning Approach](https://mlsanity.com/paper/2207.05779/) (2022)
- [One-Shot Learning of Stochastic Differential Equations with Data Adapted Kernels](https://mlsanity.com/paper/2209.12086/) (2022)
- [Learning Dynamical Systems from Data: A Simple Cross-Validation Perspective, Part V: Sparse Kernel Flows for 132 Chaotic Dynamical Systems](https://mlsanity.com/paper/2301.10321/) (2023)
- [Learning dynamical systems from data: A simple cross-validation perspective, part III: Irregularly-Sampled Time Series](https://mlsanity.com/paper/2111.13037/) (2021)
- [Solving High Frequency and Multi-Scale PDEs with Gaussian Processes](https://mlsanity.com/paper/2311.04465/) (2023)
- [Probabilistic Numerical Method of Lines for Time-Dependent Partial Differential Equations](https://mlsanity.com/paper/2110.11847/) (2021)
- [Minimax Optimal Kernel Operator Learning via Multilevel Training](https://mlsanity.com/paper/2209.14430/) (2022)
- [Learning About Structural Errors in Models of Complex Dynamical Systems](https://mlsanity.com/paper/2401.00035/) (2023)
- [Fisher-Rao Gradient Flow: Geodesic Convexity and Functional Inequalities](https://mlsanity.com/paper/2407.15693/) (2024)
- [Linearization Turns Neural Operators into Function-Valued Gaussian Processes](https://mlsanity.com/paper/2406.05072/) (2024)

## Abstract (arXiv)

We introduce a simple, rigorous, and unified framework for solving nonlinear partial differential equations (PDEs), and for solving inverse problems (IPs) involving the identification of parameters in PDEs, using the framework of Gaussian processes. The proposed approach: (1) provides a natural generalization of collocation kernel methods to nonlinear PDEs and IPs; (2) has guaranteed convergence for a very general class of PDEs, and comes equipped with a path to compute error bounds for specific PDE approximations; (3) inherits the state-of-the-art computational complexity of linear solvers for dense kernel matrices. The main idea of our method is to approximate the solution of a given PDE as the maximum a posteriori (MAP) estimator of a Gaussian process conditioned on solving the PDE at a finite number of collocation points. Although this optimization problem is infinite-dimensional, it can be reduced to a finite-dimensional one by introducing additional variables corresponding to the values of the derivatives of the solution at collocation points; this generalizes the representer theorem arising in Gaussian process regression. The reduced optimization problem has the form of a quadratic objective function subject to nonlinear constraints; it is solved with a variant of the Gauss--Newton method. The resulting algorithm (a) can be interpreted as solving successive linearizations of the nonlinear PDE, and (b) in practice is found to converge in a small number of iterations (2 to 10), for a wide range of PDEs. Most traditional approaches to IPs interleave parameter updates with numerical solution of the PDE; our algorithm solves for both parameter and PDE solution simultaneously. Experiments on nonlinear elliptic PDEs, Burgers' equation, a regularized Eikonal equation, and an IP for permeability identification in Darcy flow illustrate the efficacy and scope of our framework.