# MultiAdam: Parameter-wise Scale-invariant Optimizer for Multiscale Training of Physics-informed Neural Networks

arXiv: https://arxiv.org/abs/2306.02816 · HTML: https://mlsanity.com/paper/2306.02816/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 32 references that are in the map.
- First posted on arXiv 5 Jun 2023.

## Builds on (most cited first, 13 of 13)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks](https://mlsanity.com/paper/1711.02257/) (2017)
- [When and why PINNs fail to train: A neural tangent kernel perspective](https://mlsanity.com/paper/2007.14527/) (2020)
- [DeepXDE: A deep learning library for solving differential equations](https://mlsanity.com/paper/1907.04502/) (2019)
- [Physics-Constrained Deep Learning for High-dimensional Surrogate Modeling and Uncertainty Quantification without Labeled Data](https://mlsanity.com/paper/1901.06314/) (2019)
- [Self-Adaptive Physics-Informed Neural Networks using a Soft Attention Mechanism](https://mlsanity.com/paper/2009.04544/) (2020)
- [Solving Allen-Cahn and Cahn-Hilliard Equations using the Adaptive Physics Informed Neural Networks](https://mlsanity.com/paper/2007.04542/) (2020)
- [Physics-Informed Machine Learning: A Survey on Problems, Methods and Applications](https://mlsanity.com/paper/2211.08064/) (2022)
- [Error estimates for physics informed neural networks approximating the Navier-Stokes equations](https://mlsanity.com/paper/2203.09346/) (2022)
- [Error analysis for physics informed neural networks (PINNs) approximating Kolmogorov PDEs](https://mlsanity.com/paper/2106.14473/) (2021)
- [Accelerating Physics-Informed Neural Network Training with Prior Dictionaries](https://mlsanity.com/paper/2004.08151/) (2020)
- [CoPhy-PGNN: Learning Physics-guided Neural Networks with Competing Loss Functions for Solving Eigenvalue Problems](https://mlsanity.com/paper/2007.01420/) (2020)

## Built on by (most cited first, 17 of 17)

- [PINNacle: A Comprehensive Benchmark of Physics-Informed Neural Networks for Solving PDEs](https://mlsanity.com/paper/2306.08827/) (2023)
- [ConFIG: Towards Conflict-free Training of Physics Informed Neural Networks](https://mlsanity.com/paper/2408.11104/) (2024)
- [RoPINN: Region Optimized Physics-Informed Neural Networks](https://mlsanity.com/paper/2405.14369/) (2024)
- [Dual Cone Gradient Descent for Training Physics-Informed Neural Networks](https://mlsanity.com/paper/2409.18426/) (2024)
- [Enhancing Physics-Informed Neural Networks Through Feature Engineering](https://mlsanity.com/paper/2502.07209/) (2025)
- [Robust Weight Initialization for Tanh Neural Networks with Fixed Point Analysis](https://mlsanity.com/paper/2410.02242/) (2024)
- [Physics-informed Neural Networks for Functional Differential Equations: Cylindrical Approximation and Its Convergence Guarantees](https://mlsanity.com/paper/2410.18153/) (2024)
- [Advancing Generalization in PINNs through Latent-Space Representations](https://mlsanity.com/paper/2411.19125/) (2024)
- [EvoPINN: Agentic Discovery of Executable Algorithms for Physics-Informed Neural Networks](https://mlsanity.com/paper/2607.26490/) (2026)
- [Tackling Failure Modes of PINNs and PIKANs Using Conflict-Free Gradients](https://mlsanity.com/paper/2609.14841/) (2026)
- [Sinc Kolmogorov-Arnold network and its application for solving PDEs with singularities](https://mlsanity.com/paper/2410.04096/) (2024)
- [Enhancing Stability of Physics-Informed Neural Network Training Through Saddle-Point Reformulation](https://mlsanity.com/paper/2507.16008/) (2025)
- [Equation identification for fluid flows via physics-informed neural networks](https://mlsanity.com/paper/2408.17271/) (2024)
- [MUSA-PINN: Multi-scale Weak-form Physics-Informed Neural Networks for Fluid Flow in Complex Geometries](https://mlsanity.com/paper/2603.08465/) (2026)
- [Conflict-Aware Harmonized Rotational Gradient for Multiscale Kinetic Regimes](https://mlsanity.com/paper/2604.24745/) (2026)
- [Chebyshev Center-Based Direction Selection for Multi-Objective Optimization and Training PINNs](https://mlsanity.com/paper/2605.09975/) (2026)
- [Modularity-Free Conflict-Averse Training for Generalized PINNs](https://mlsanity.com/paper/2606.20156/) (2026)

## Abstract (arXiv)

Physics-informed Neural Networks (PINNs) have recently achieved remarkable progress in solving Partial Differential Equations (PDEs) in various fields by minimizing a weighted sum of PDE loss and boundary loss. However, there are several critical challenges in the training of PINNs, including the lack of theoretical frameworks and the imbalance between PDE loss and boundary loss. In this paper, we present an analysis of second-order non-homogeneous PDEs, which are classified into three categories and applicable to various common problems. We also characterize the connections between the training loss and actual error, guaranteeing convergence under mild conditions. The theoretical analysis inspires us to further propose MultiAdam, a scale-invariant optimizer that leverages gradient momentum to parameter-wisely balance the loss terms. Extensive experiment results on multiple problems from different physical domains demonstrate that our MultiAdam solver can improve the predictive accuracy by 1-2 orders of magnitude compared with strong baselines.