# Improving physics-informed neural networks with meta-learned optimization

arXiv: https://arxiv.org/abs/2303.07127 · HTML: https://mlsanity.com/paper/2303.07127/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 42 references that are in the map.
- First posted on arXiv 13 Mar 2023.

## Builds on (most cited first, 27 of 27)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [Automatic differentiation in machine learning: a survey](https://mlsanity.com/paper/1502.05767/) (2015)
- [Multi-Task Learning as Multi-Objective Optimization](https://mlsanity.com/paper/1810.04650/) (2018)
- [Learning to learn by gradient descent by gradient descent](https://mlsanity.com/paper/1606.04474/) (2016)
- [Scientific Machine Learning through Physics-Informed Neural Networks: Where we are and What's next](https://mlsanity.com/paper/2201.05624/) (2022)
- [hp-VPINNs: Variational Physics-Informed Neural Networks With Domain Decomposition](https://mlsanity.com/paper/2003.05385/) (2020)
- [A comprehensive study of non-adaptive and residual-based adaptive sampling for physics-informed neural networks](https://mlsanity.com/paper/2207.10289/) (2022)
- [Learning to Optimize: A Primer and A Benchmark](https://mlsanity.com/paper/2103.12828/) (2021)
- [Respecting causality is all you need for training physics-informed neural networks](https://mlsanity.com/paper/2203.07404/) (2022)
- [Learned Optimizers that Scale and Generalize](https://mlsanity.com/paper/1703.04813/) (2017)
- [On Empirical Comparisons of Optimizers for Deep Learning](https://mlsanity.com/paper/1910.05446/) (2019)
- [Long-time integration of parametric evolution equations with physics-informed DeepONets](https://mlsanity.com/paper/2106.05384/) (2021)
- [Unbiased Gradient Estimation in Unrolled Computation Graphs with Persistent Evolution Strategies](https://mlsanity.com/paper/2112.13835/) (2021)
- [Meta-learning PINN loss functions](https://mlsanity.com/paper/2107.05544/) (2021)
- [Learning Gradient Descent: Better Generalization and Longer Horizons](https://mlsanity.com/paper/1703.03633/) (2017)
- [A unified scalable framework for causal sweeping strategies for Physics-Informed Neural Networks (PINNs) and their temporal decompositions](https://mlsanity.com/paper/2302.14227/) (2023)
- [Aggregated Momentum: Stability Through Passive Damping](https://mlsanity.com/paper/1804.00325/) (2018)
- [A novel meta-learning initialization method for physics-informed neural networks](https://mlsanity.com/paper/2107.10991/) (2021)
- [Physics-informed neural networks for the shallow-water equations on the sphere](https://mlsanity.com/paper/2104.00615/) (2021)
- [Practical tradeoffs between memory, compute, and performance in learned optimizers](https://mlsanity.com/paper/2203.11860/) (2022)
- [A Closer Look at Learned Optimization: Stability, Robustness, and Inductive Biases](https://mlsanity.com/paper/2209.11208/) (2022)
- [Reverse engineering learned optimizers reveals known and novel mechanisms](https://mlsanity.com/paper/2011.02159/) (2020)

## Built on by (most cited first, 19 of 19)

- [Unveiling the optimization process of Physics Informed Neural Networks: How accurate and competitive can PINNs be?](https://mlsanity.com/paper/2405.04230/) (2024)
- [Exactly conservative physics-informed neural networks and deep operator networks for dynamical systems](https://mlsanity.com/paper/2311.14131/) (2023)
- [Evolutionary Optimization of Physics-Informed Neural Networks: Evo-PINN Frontiers and Opportunities](https://mlsanity.com/paper/2501.06572/) (2025)
- [Learning a Neural Solver for Parametric PDE to Enhance Physics-Informed Methods](https://mlsanity.com/paper/2410.06820/) (2024)
- [Improving physics-informed DeepONets with hard constraints](https://mlsanity.com/paper/2309.07899/) (2023)
- [Out-of-Distribution Generalization for Neural Physics Solvers](https://mlsanity.com/paper/2601.19091/) (2026)
- [M-ENIAC: A machine learning recreation of the first successful numerical weather forecasts](https://mlsanity.com/paper/2304.09070/) (2023)
- [A Sequential Meta-Transfer (SMT) Learning to Combat Complexities of Physics-Informed Neural Networks: Application to Composites Autoclave Processing](https://mlsanity.com/paper/2308.06447/) (2023)
- [Gabor-Enhanced Physics-Informed Neural Networks for Fast Simulations of Acoustic Wavefields](https://mlsanity.com/paper/2502.17134/) (2025)
- [TINNs: Time-Induced Neural Networks for Solving Time-Dependent PDEs](https://mlsanity.com/paper/2601.20361/) (2026)
- [Convergence of Stochastic Gradient Methods for Wide Two-Layer Physics-Informed Neural Networks for the Poisson Equation](https://mlsanity.com/paper/2508.21571/) (2025)
- [PinnDE: Physics-Informed Neural Networks for Solving Differential Equations](https://mlsanity.com/paper/2408.10011/) (2024)
- [Solving 2-D Helmholtz equation in the rectangular, circular, and elliptical domains using neural networks](https://mlsanity.com/paper/2503.20222/) (2025)
- [Architectural Strategies for the optimization of Physics-Informed Neural Networks](https://mlsanity.com/paper/2402.02711/) (2024)
- [Least-Squares-Embedded Optimization for Accelerated Convergence of PINNs in Acoustic Wavefield Simulations](https://mlsanity.com/paper/2504.16553/) (2025)
- [Latent Representation Learning of Multi-scale Thermophysics: Application to Dynamics in Shocked Porous Energetic Material](https://mlsanity.com/paper/2506.12996/) (2025)
- [Low-rank adaptive physics-informed HyperDeepONets for solving differential equations](https://mlsanity.com/paper/2507.18346/) (2025)
- [Intrinsic-Metric Physics-Informed Neural Networks (IM-PINN) for Reaction-Diffusion Dynamics on Complex Riemannian Manifolds](https://mlsanity.com/paper/2601.00834/) (2025)
- [A Comparative Investigation of Thermodynamic Structure-Informed Neural Networks](https://mlsanity.com/paper/2603.26803/) (2026)

## Abstract (arXiv)

We show that the error achievable using physics-informed neural networks for solving systems of differential equations can be substantially reduced when these networks are trained using meta-learned optimization methods rather than to using fixed, hand-crafted optimizers as traditionally done. We choose a learnable optimization method based on a shallow multi-layer perceptron that is meta-trained for specific classes of differential equations. We illustrate meta-trained optimizers for several equations of practical relevance in mathematical physics, including the linear advection equation, Poisson's equation, the Korteweg--de Vries equation and Burgers' equation. We also illustrate that meta-learned optimizers exhibit transfer learning abilities, in that a meta-trained optimizer on one differential equation can also be successfully deployed on another differential equation.