# Meta-learning PINN loss functions

arXiv: https://arxiv.org/abs/2107.05544 · HTML: https://mlsanity.com/paper/2107.05544/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 32 references that are in the map.
- First posted on arXiv 12 Jul 2021.

## Builds on (most cited first, 24 of 24)

- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [On First-Order Meta-Learning Algorithms](https://mlsanity.com/paper/1803.02999/) (2018)
- [Meta-Learning in Neural Networks: A Survey](https://mlsanity.com/paper/2004.05439/) (2020)
- [DeepXDE: A deep learning library for solving differential equations](https://mlsanity.com/paper/1907.04502/) (2019)
- [Meta-Learning with Implicit Gradients](https://mlsanity.com/paper/1909.04630/) (2019)
- [B-PINNs: Bayesian Physics-Informed Neural Networks for Forward and Inverse PDE Problems with Noisy Data](https://mlsanity.com/paper/2003.06097/) (2020)
- [Optimizing Millions of Hyperparameters by Implicit Differentiation](https://mlsanity.com/paper/1911.02590/) (2019)
- [A General and Adaptive Robust Loss Function](https://mlsanity.com/paper/1701.03077/) (2017)
- [hp-VPINNs: Variational Physics-Informed Neural Networks With Domain Decomposition](https://mlsanity.com/paper/2003.05385/) (2020)
- [Parallel Physics-Informed Neural Networks via Domain Decomposition](https://mlsanity.com/paper/2104.10013/) (2021)
- [On Learning Intrinsic Rewards for Policy Gradient Methods](https://mlsanity.com/paper/1804.06459/) (2018)
- [Dynamics of Deep Neural Networks and Neural Tangent Hierarchy](https://mlsanity.com/paper/1909.08156/) (2019)
- [Generalized Inner Loop Meta-Learning](https://mlsanity.com/paper/1910.01727/) (2019)
- [Evolved Policy Gradients](https://mlsanity.com/paper/1802.04821/) (2018)
- [Meta-Learning via Learned Loss](https://mlsanity.com/paper/1906.05374/) (2019)
- [Learning to Teach with Dynamic Loss Functions](https://mlsanity.com/paper/1810.12081/) (2018)
- [nPINNs: nonlocal Physics-Informed Neural Networks for a parametrized nonlocal universal Laplacian operator. Algorithms and Applications](https://mlsanity.com/paper/2004.04276/) (2020)
- [Addressing the Loss-Metric Mismatch with Adaptive Loss Alignment](https://mlsanity.com/paper/1905.05895/) (2019)
- [Reward Shaping via Meta-Learning](https://mlsanity.com/paper/1901.09330/) (2019)
- [Gradient Descent Finds Global Minima for Generalizable Deep Neural Networks of Practical Sizes](https://mlsanity.com/paper/1908.02419/) (2019)
- [Learning to Learn: Meta-Critic Networks for Sample Efficient Learning](https://mlsanity.com/paper/1706.09529/) (2017)
- [Learning Surrogate Losses](https://mlsanity.com/paper/1905.10108/) (2019)
- [A Recipe for Global Convergence Guarantee in Deep Neural Networks](https://mlsanity.com/paper/2104.05785/) (2021)
- [Learning to learn via Self-Critique](https://mlsanity.com/paper/1905.10295/) (2019)

## Built on by (most cited first, 30 of 51)

- [A comprehensive study of non-adaptive and residual-based adaptive sampling for physics-informed neural networks](https://mlsanity.com/paper/2207.10289/) (2022)
- [Uncertainty Quantification in Scientific Machine Learning: Methods, Metrics, and Comparisons](https://mlsanity.com/paper/2201.07766/) (2022)
- [Physics-Informed Machine Learning: A Survey on Problems, Methods and Applications](https://mlsanity.com/paper/2211.08064/) (2022)
- [A Metalearning Approach for Physics-Informed Neural Networks (PINNs): Application to Parameterized PDEs](https://mlsanity.com/paper/2110.13361/) (2021)
- [Super-Resolution Analysis via Machine Learning: A Survey for Fluid Flows](https://mlsanity.com/paper/2301.10937/) (2023)
- [Hutchinson Trace Estimation for High-Dimensional and High-Order Physics-Informed Neural Networks](https://mlsanity.com/paper/2312.14499/) (2023)
- [GPT-PINN: Generative Pre-Trained Physics-Informed Neural Networks toward non-intrusive Meta-learning of parametric PDEs](https://mlsanity.com/paper/2303.14878/) (2023)
- [Improving physics-informed neural networks with meta-learned optimization](https://mlsanity.com/paper/2303.07127/) (2023)
- [Meta-PDE: Learning to Solve PDEs Quickly Without a Mesh](https://mlsanity.com/paper/2211.01604/) (2022)
- [HANNA: Hard-constraint Neural Network for Consistent Activity Coefficient Prediction](https://mlsanity.com/paper/2407.18011/) (2024)
- [Learning Specialized Activation Functions for Physics-informed Neural Networks](https://mlsanity.com/paper/2308.04073/) (2023)
- [Physics-Informed Deep Learning For Traffic State Estimation: A Survey and the Outlook](https://mlsanity.com/paper/2303.02063/) (2023)
- [Physics-aware Machine Learning Revolutionizes Scientific Paradigm for Machine Learning and Process-based Hydrology](https://mlsanity.com/paper/2310.05227/) (2023)
- [Evolutionary Optimization of Physics-Informed Neural Networks: Evo-PINN Frontiers and Opportunities](https://mlsanity.com/paper/2501.06572/) (2025)
- [Training multi-objective/multi-task collocation physics-informed neural network with student/teachers transfer learnings](https://mlsanity.com/paper/2107.11496/) (2021)
- [Learning Symbolic Model-Agnostic Loss Functions via Meta-Learning](https://mlsanity.com/paper/2209.08907/) (2022)
- [Federated scientific machine learning for approximating functions and solving differential equations with data heterogeneity](https://mlsanity.com/paper/2410.13141/) (2024)
- [Principled Acceleration of Iterative Numerical Methods Using Machine Learning](https://mlsanity.com/paper/2206.08594/) (2022)
- [PINN Training using Biobjective Optimization: The Trade-off between Data Loss and Residual Loss](https://mlsanity.com/paper/2302.01810/) (2023)
- [Energy Dissipation Preserving Physics Informed Neural Network for Allen-Cahn Equations](https://mlsanity.com/paper/2411.08760/) (2024)
- [Learning PDE Solvers with Physics and Data: A Unifying View of Physics-Informed Neural Networks and Neural Operators](https://mlsanity.com/paper/2601.14517/) (2026)
- [Deep Random Vortex Method for Simulation and Inference of Navier-Stokes Equations](https://mlsanity.com/paper/2206.09571/) (2022)
- [Adaptive Physics-informed Neural Networks: A Survey](https://mlsanity.com/paper/2503.18181/) (2025)
- [Out-of-Distribution Generalization for Neural Physics Solvers](https://mlsanity.com/paper/2601.19091/) (2026)
- [Residual-Quantile Adjustment for Adaptive Training of Physics-informed Neural Network](https://mlsanity.com/paper/2209.05315/) (2022)
- [Meta-Learning Adaptive Loss Functions](https://mlsanity.com/paper/2301.13247/) (2023)
- [Machine Learning in Heterogeneous Porous Materials](https://mlsanity.com/paper/2202.04137/) (2022)
- [A Control Perspective on Training PINNs](https://mlsanity.com/paper/2501.18582/) (2025)
- [iPINNER: An Iterative Physics-Informed Neural Network with Ensemble Kalman Filter](https://mlsanity.com/paper/2506.00731/) (2025)
- [Feature Mapping in Physics-Informed Neural Networks (PINNs)](https://mlsanity.com/paper/2402.06955/) (2024)

## Abstract (arXiv)

We propose a meta-learning technique for offline discovery of physics-informed neural network (PINN) loss functions. We extend earlier works on meta-learning, and develop a gradient-based meta-learning algorithm for addressing diverse task distributions based on parametrized partial differential equations (PDEs) that are solved with PINNs. Furthermore, based on new theory we identify two desirable properties of meta-learned losses in PINN problems, which we enforce by proposing a new regularization method or using a specific parametrization of the loss function. In the computational examples, the meta-learned losses are employed at test time for addressing regression and PDE task distributions. Our results indicate that significant performance improvement can be achieved by using a shared-among-tasks offline-learned loss function even for out-of-distribution meta-testing. In this case, we solve for test tasks that do not belong to the task distribution used in meta-training, and we also employ PINN architectures that are different from the PINN architecture used in meta-training. To better understand the capabilities and limitations of the proposed method, we consider various parametrizations of the loss function and describe different algorithm design options and how they may affect meta-learning performance.