# Architectural Strategies for the optimization of Physics-Informed Neural Networks

arXiv: https://arxiv.org/abs/2402.02711 · HTML: https://mlsanity.com/paper/2402.02711/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 53 references that are in the map.
- First posted on arXiv 5 Feb 2024.

## Builds on (most cited first, 29 of 29)

- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [A Convergence Theory for Deep Learning via Over-Parameterization](https://mlsanity.com/paper/1811.03962/) (2018)
- [Gradient Descent Provably Optimizes Over-parameterized Neural Networks](https://mlsanity.com/paper/1810.02054/) (2018)
- [When and why PINNs fail to train: A neural tangent kernel perspective](https://mlsanity.com/paper/2007.14527/) (2020)
- [Scientific Machine Learning through Physics-Informed Neural Networks: Where we are and What's next](https://mlsanity.com/paper/2201.05624/) (2022)
- [Lipschitz regularity of deep neural networks: analysis and efficient estimation](https://mlsanity.com/paper/1805.10965/) (2018)
- [B-PINNs: Bayesian Physics-Informed Neural Networks for Forward and Inverse PDE Problems with Noisy Data](https://mlsanity.com/paper/2003.06097/) (2020)
- [On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks](https://mlsanity.com/paper/2012.10047/) (2020)
- [WIRE: Wavelet Implicit Neural Representations](https://mlsanity.com/paper/2301.05187/) (2023)
- [Towards moderate overparameterization: global convergence guarantees for training shallow neural networks](https://mlsanity.com/paper/1902.04674/) (2019)
- [Understanding and mitigating gradient pathologies in physics-informed neural networks](https://mlsanity.com/paper/2001.04536/) (2020)
- [Beyond Periodicity: Towards a Unifying Framework for Activations in Coordinate-MLPs](https://mlsanity.com/paper/2111.15135/) (2021)
- [Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU Networks](https://mlsanity.com/paper/2012.11654/) (2020)
- [Multi-Objective Loss Balancing for Physics-Informed Deep Learning](https://mlsanity.com/paper/2110.09813/) (2021)
- [GARF: Gaussian Activated Radiance Fields for High Fidelity Reconstruction and Pose Estimation](https://mlsanity.com/paper/2204.05735/) (2022)
- [Overview frequency principle/spectral bias in deep learning](https://mlsanity.com/paper/2201.07395/) (2022)
- [Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessians](https://mlsanity.com/paper/1901.08244/) (2019)
- [Global Convergence of Deep Networks with One Wide Layer Followed by Pyramidal Topology](https://mlsanity.com/paper/2002.07867/) (2020)
- [Learning in Modal Space: Solving Time-Dependent Stochastic PDEs Using Physics-Informed Neural Networks](https://mlsanity.com/paper/1905.01205/) (2019)
- [PINNsFormer: A Transformer-Based Framework For Physics-Informed Neural Networks](https://mlsanity.com/paper/2307.11833/) (2023)
- [Memorization and Optimization in Deep Neural Networks with Minimum Over-parameterization](https://mlsanity.com/paper/2205.10217/) (2022)
- [Improving physics-informed neural networks with meta-learned optimization](https://mlsanity.com/paper/2303.07127/) (2023)
- [Critical Investigation of Failure Modes in Physics-informed Neural Networks](https://mlsanity.com/paper/2206.09961/) (2022)
- [Physics-informed Neural Networks with Periodic Activation Functions for Solute Transport in Heterogeneous Porous Media](https://mlsanity.com/paper/2212.08965/) (2022)
- [Curvature-Aware Training for Coordinate Networks](https://mlsanity.com/paper/2305.08552/) (2023)
- [Probing optimisation in physics-informed neural networks](https://mlsanity.com/paper/2303.15196/) (2023)

## Abstract (arXiv)

Physics-informed neural networks (PINNs) offer a promising avenue for tackling both forward and inverse problems in partial differential equations (PDEs) by incorporating deep learning with fundamental physics principles. Despite their remarkable empirical success, PINNs have garnered a reputation for their notorious training challenges across a spectrum of PDEs. In this work, we delve into the intricacies of PINN optimization from a neural architecture perspective. Leveraging the Neural Tangent Kernel (NTK), our study reveals that Gaussian activations surpass several alternate activations when it comes to effectively training PINNs. Building on insights from numerical linear algebra, we introduce a preconditioned neural architecture, showcasing how such tailored architectures enhance the optimization process. Our theoretical findings are substantiated through rigorous validation against established PDEs within the scientific literature.