# Reinforcement Learning Closures for Underresolved Partial Differential Equations using Synthetic Data

arXiv: https://arxiv.org/abs/2505.11308 · HTML: https://mlsanity.com/paper/2505.11308/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 25 of its 51 references that are in the map.
- First posted on arXiv 16 May 2025.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 25 of 25)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [Hamiltonian Neural Networks](https://mlsanity.com/paper/1906.01563/) (2019)
- [Learning the solution operator of parametric partial differential equations with physics-informed DeepOnets](https://mlsanity.com/paper/2103.10974/) (2021)
- [Physics-informed neural networks (PINNs) for fluid mechanics: A review](https://mlsanity.com/paper/2105.09506/) (2021)
- [Learning Deep CNN Denoiser Prior for Image Restoration](https://mlsanity.com/paper/1704.03264/) (2017)
- [Lagrangian Neural Networks](https://mlsanity.com/paper/2003.04630/) (2020)
- [Machine Learning for Fluid Mechanics](https://mlsanity.com/paper/1905.11075/) (2019)
- [Deep Lagrangian Networks: Using Physics as Model Prior for Deep Learning](https://mlsanity.com/paper/1907.04490/) (2019)
- [The Well: a Large-Scale Collection of Diverse Physics Simulations for Machine Learning](https://mlsanity.com/paper/2412.00568/) (2024)
- [Tianshou: a Highly Modularized Deep Reinforcement Learning Library](https://mlsanity.com/paper/2107.14171/) (2021)
- [Hamiltonian Generative Networks](https://mlsanity.com/paper/1909.13789/) (2019)
- [Multiscale Simulations of Complex Systems by Learning their Effective Dynamics](https://mlsanity.com/paper/2006.13431/) (2020)
- [Scientific multi-agent reinforcement learning for wall-models of turbulent flows](https://mlsanity.com/paper/2106.11144/) (2021)
- [A probabilistic generative model for semi-supervised training of coarse-grained surrogates and enforcing physical constraints through virtual observables](https://mlsanity.com/paper/2006.01789/) (2020)
- [Incorporating physical constraints in a deep probabilistic machine learning framework for coarse-graining dynamical systems](https://mlsanity.com/paper/1912.12976/) (2019)
- [Semi-supervised Invertible Neural Operators for Bayesian Inverse Problems](https://mlsanity.com/paper/2209.02772/) (2022)
- [Deep neural network enabled corrective source term approach to hybrid analysis and modeling](https://mlsanity.com/paper/2105.11521/) (2021)
- [Physics-aware, probabilistic model order reduction with guaranteed stability](https://mlsanity.com/paper/2101.05834/) (2021)
- [Generating synthetic data for neural operators](https://mlsanity.com/paper/2401.02398/) (2024)
- [Weak neural variational inference for solving Bayesian inverse problems without forward models: applications in elastography](https://mlsanity.com/paper/2407.20697/) (2024)
- [DGenNO: A Novel Physics-aware Neural Operator for Solving Forward and Inverse PDE Problems based on Deep, Generative Probabilistic Modeling](https://mlsanity.com/paper/2502.06250/) (2025)
- [Closure Discovery for Coarse-Grained Partial Differential Equations Using Grid-based Reinforcement Learning](https://mlsanity.com/paper/2402.00972/) (2024)

## Abstract (arXiv)

Partial Differential Equations (PDEs) describe phenomena ranging from turbulence and epidemics to quantum mechanics and financial markets. Despite recent advances in computational science, solving such PDEs for real-world applications remains prohibitively expensive because of the necessity of resolving a broad range of spatiotemporal scales. In turn, practitioners often rely on coarse-grained approximations of the original PDEs, trading off accuracy for reduced computational resources. To mitigate the loss of detail inherent in such approximations, closure models are employed to represent unresolved spatiotemporal interactions. We present a framework for developing closure models for PDEs using synthetic data acquired through the method of manufactured solutions. These data are used in conjunction with reinforcement learning to provide closures for coarse-grained PDEs. We illustrate the efficacy of our method using the one-dimensional and two-dimensional Burgers' equations and the two-dimensional advection equation. Moreover, we demonstrate that closure models trained for inhomogeneous PDEs can be effectively generalized to homogeneous PDEs. The results demonstrate the potential for developing accurate and computationally efficient closure models for systems with scarce data.