# PDE-constrained Models with Neural Network Terms: Optimization and Global Convergence

arXiv: https://arxiv.org/abs/2105.08633 · HTML: https://mlsanity.com/paper/2105.08633/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 72 references that are in the map.
- First posted on arXiv 18 May 2021.

## Builds on (most cited first, 10 of 10)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Machine learning accelerated computational fluid dynamics](https://mlsanity.com/paper/2102.01010/) (2021)
- [Machine Learning for Fluid Mechanics](https://mlsanity.com/paper/1905.11075/) (2019)
- [B-PINNs: Bayesian Physics-Informed Neural Networks for Forward and Inverse PDE Problems with Noisy Data](https://mlsanity.com/paper/2003.06097/) (2020)
- [hp-VPINNs: Variational Physics-Informed Neural Networks With Domain Decomposition](https://mlsanity.com/paper/2003.05385/) (2020)
- [Deep Learning of Vortex Induced Vibrations](https://mlsanity.com/paper/1808.08952/) (2018)
- [DPM: A deep learning PDE augmentation method (with application to large-eddy simulation)](https://mlsanity.com/paper/1911.09145/) (2019)
- [Embedded training of neural-network sub-grid-scale turbulence models](https://mlsanity.com/paper/2105.01030/) (2021)
- [Neural network augmented inverse problems for PDEs](https://mlsanity.com/paper/1712.09685/) (2017)
- [Simultaneous Single-Step One-Shot Optimization with Unsteady PDEs](https://mlsanity.com/paper/1503.00884/) (2015)

## Built on by (most cited first, 12 of 12)

- [Deep Learning Closure Models for Large-Eddy Simulation of Flows around Bluff Bodies](https://mlsanity.com/paper/2208.03498/) (2022)
- [Neural Q-learning for solving PDEs](https://mlsanity.com/paper/2203.17128/) (2022)
- [PICT -- A Differentiable, GPU-Accelerated Multi-Block PISO Solver for Simulation-Coupled Learning Tasks in Fluid Dynamics](https://mlsanity.com/paper/2505.16992/) (2025)
- [An Efficient High-Dimensional Gradient Estimator for Stochastic Differential Equations](https://mlsanity.com/paper/2407.10065/) (2024)
- [oRANS: Online optimisation of RANS machine learning models with embedded DNS data generation](https://mlsanity.com/paper/2510.02982/) (2025)
- [Learning-based Multi-continuum Model for Multiscale Flow Problems](https://mlsanity.com/paper/2403.14084/) (2024)
- [OGF: An Online Gradient Flow Method for Optimizing the Statistical Steady-State Time Averages of Unsteady Turbulent Flows](https://mlsanity.com/paper/2507.05149/) (2025)
- [Physics-Based Machine Learning Closures and Wall Models for Hypersonic Transition-Continuum Boundary Layer Predictions](https://mlsanity.com/paper/2507.08986/) (2025)
- [Active Control of Turbulent Airfoil Flows Using Adjoint-based Deep Learning](https://mlsanity.com/paper/2510.07106/) (2025)
- [Symbolic recovery of PDEs from measurement data](https://mlsanity.com/paper/2602.15603/) (2026)
- [Convergence Analysis of Newton's Method for Neural Networks in the Overparameterized Limit](https://mlsanity.com/paper/2605.08352/) (2026)
- [Deep Learning-based Algebraic Reynolds Stress Closures for RANS Simulations of Turbulent Flows](https://mlsanity.com/paper/2605.26358/) (2026)

## Abstract (arXiv)

Recent research has used deep learning to develop partial differential equation (PDE) models in science and engineering. The functional form of the PDE is determined by a neural network, and the neural network parameters are calibrated to available data. Calibration of the embedded neural network can be performed by optimizing over the PDE. Motivated by these applications, we rigorously study the optimization of a class of linear elliptic PDEs with neural network terms. The neural network parameters in the PDE are optimized using gradient descent, where the gradient is evaluated using an adjoint PDE. As the number of parameters become large, the PDE and adjoint PDE converge to a non-local PDE system. Using this limit PDE system, we are able to prove convergence of the neural network-PDE to a global minimum during the optimization. Finally, we use this adjoint method to train a neural network model for an application in fluid mechanics, in which the neural network functions as a closure model for the Reynolds-averaged Navier--Stokes (RANS) equations. The RANS neural network model is trained on several datasets for turbulent channel flow and is evaluated out-of-sample at different Reynolds numbers.