# Inverse-Dirichlet Weighting Enables Reliable Training of Physics Informed Neural Networks

arXiv: https://arxiv.org/abs/2107.00940 · HTML: https://mlsanity.com/paper/2107.00940/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 58 references that are in the map.
- First posted on arXiv 2 Jul 2021.

## Builds on (most cited first, 28 of 28)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks](https://mlsanity.com/paper/1711.02257/) (2017)
- [Multi-Task Learning as Multi-Objective Optimization](https://mlsanity.com/paper/1810.04650/) (2018)
- [DGM: A deep learning algorithm for solving partial differential equations](https://mlsanity.com/paper/1708.07469/) (2017)
- [When and why PINNs fail to train: A neural tangent kernel perspective](https://mlsanity.com/paper/2007.14527/) (2020)
- [Solving high-dimensional partial differential equations using deep learning](https://mlsanity.com/paper/1707.02568/) (2017)
- [Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks](https://mlsanity.com/paper/1901.06523/) (2019)
- [B-PINNs: Bayesian Physics-Informed Neural Networks for Forward and Inverse PDE Problems with Noisy Data](https://mlsanity.com/paper/2003.06097/) (2020)
- [Stochastic Gradient Descent as Approximate Bayesian Inference](https://mlsanity.com/paper/1704.04289/) (2017)
- [Training behavior of deep neural network in frequency domain](https://mlsanity.com/paper/1807.01251/) (2018)
- [Sobolev Training for Neural Networks](https://mlsanity.com/paper/1706.04859/) (2017)
- [Understanding and mitigating gradient pathologies in physics-informed neural networks](https://mlsanity.com/paper/2001.04536/) (2020)
- [Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers](https://mlsanity.com/paper/2007.01547/) (2020)
- [Error bounds for approximations with deep ReLU neural networks in Ws,p norms](https://mlsanity.com/paper/1902.07896/) (2019)
- [Physics informed deep learning for computational elastodynamics without labeled data](https://mlsanity.com/paper/2006.08472/) (2020)
- [Deep Learning of Vortex Induced Vibrations](https://mlsanity.com/paper/1808.08952/) (2018)
- [DeepMoD: Deep learning for Model Discovery in noisy data](https://mlsanity.com/paper/1904.09406/) (2019)
- [Learning in Modal Space: Solving Time-Dependent Stochastic PDEs Using Physics-Informed Neural Networks](https://mlsanity.com/paper/1905.01205/) (2019)
- [Understanding training and generalization in deep learning by Fourier analysis](https://mlsanity.com/paper/1808.04295/) (2018)
- [DenoiSeg: Joint Denoising and Segmentation](https://mlsanity.com/paper/2005.02987/) (2020)
- [Deep Learning of Turbulent Scalar Mixing](https://mlsanity.com/paper/1811.07095/) (2018)
- [Physics-informed semantic inpainting: Application to geostatistical modeling](https://mlsanity.com/paper/1909.09459/) (2019)

## Built on by (most cited first, 30 of 43)

- [Respecting causality is all you need for training physics-informed neural networks](https://mlsanity.com/paper/2203.07404/) (2022)
- [An Expert's Guide to Training Physics-informed Neural Networks](https://mlsanity.com/paper/2308.08468/) (2023)
- [Physics-Informed Machine Learning: A Survey on Problems, Methods and Applications](https://mlsanity.com/paper/2211.08064/) (2022)
- [PINNacle: A Comprehensive Benchmark of Physics-Informed Neural Networks for Solving PDEs](https://mlsanity.com/paper/2306.08827/) (2023)
- [PirateNets: Physics-informed Deep Learning with Residual Adaptive Networks](https://mlsanity.com/paper/2402.00326/) (2024)
- [RoPINN: Region Optimized Physics-Informed Neural Networks](https://mlsanity.com/paper/2405.14369/) (2024)
- [On the Role of Fixed Points of Dynamical Systems in Training Physics-Informed Neural Networks](https://mlsanity.com/paper/2203.13648/) (2022)
- [Functional Tensor Decompositions for Physics-Informed Neural Networks](https://mlsanity.com/paper/2408.13101/) (2024)
- [BiLO: Bilevel Local Operator Learning for PDE Inverse Problems](https://mlsanity.com/paper/2404.17789/) (2024)
- [Personalized Predictions of Glioblastoma Infiltration: Mathematical Models, Physics-Informed Neural Networks and Multimodal Scans](https://mlsanity.com/paper/2311.16536/) (2023)
- [PINN Training using Biobjective Optimization: The Trade-off between Data Loss and Residual Loss](https://mlsanity.com/paper/2302.01810/) (2023)
- [Physics-Informed Neural Networks for Dynamic Process Operations with Limited Physical Knowledge and Data](https://mlsanity.com/paper/2406.01528/) (2024)
- [A physics-informed transformer neural operator for learning generalized solutions of initial boundary value problems](https://mlsanity.com/paper/2412.09009/) (2024)
- [Stochastic force inference via density estimation](https://mlsanity.com/paper/2310.02366/) (2023)
- [Dual-Balancing for Physics-Informed Neural Networks](https://mlsanity.com/paper/2505.11117/) (2025)
- [Tunable Complexity Benchmarks for Evaluating Physics-Informed Neural Networks on Coupled Ordinary Differential Equations](https://mlsanity.com/paper/2210.07880/) (2022)
- [Replacing Automatic Differentiation by Sobolev Cubatures fastens Physics Informed Neural Nets and strengthens their Approximation Power](https://mlsanity.com/paper/2211.15443/) (2022)
- [LSA-PINN: Linear Boundary Connectivity Loss for Solving PDEs on Complex Geometry](https://mlsanity.com/paper/2302.01518/) (2023)
- [Adapting Physics-Informed Neural Networks to Improve ODE Optimization in Mosquito Population Dynamics](https://mlsanity.com/paper/2406.05108/) (2024)
- [Solving Differential Equations with Constrained Learning](https://mlsanity.com/paper/2410.22796/) (2024)
- [A Control Perspective on Training PINNs](https://mlsanity.com/paper/2501.18582/) (2025)
- [Physics-Informed Koopman Network](https://mlsanity.com/paper/2211.09419/) (2022)
- [Common pitfalls to avoid while using multiobjective optimization in machine learning](https://mlsanity.com/paper/2405.01480/) (2024)
- [Sample-Efficient Reinforcement Learning of Koopman eNMPC](https://mlsanity.com/paper/2503.18787/) (2025)
- [Learning Partial Differential Equations by Spectral Approximates of General Sobolev Spaces](https://mlsanity.com/paper/2301.04887/) (2023)
- [Maximum-likelihood Estimators in Physics-Informed Neural Networks for High-dimensional Inverse Problems](https://mlsanity.com/paper/2304.05991/) (2023)
- [Auto-weighted Bayesian Physics-Informed Neural Networks and robust estimations for multitask inverse problems in pore-scale imaging of dissolution](https://mlsanity.com/paper/2308.12864/) (2023)
- [Solving 2-D Helmholtz equation in the rectangular, circular, and elliptical domains using neural networks](https://mlsanity.com/paper/2503.20222/) (2025)
- [Physics-Informed Neural Networks For Semiconductor Film Deposition: A Review](https://mlsanity.com/paper/2507.10983/) (2025)
- [Enhancing Stability of Physics-Informed Neural Network Training Through Saddle-Point Reformulation](https://mlsanity.com/paper/2507.16008/) (2025)

## Abstract (arXiv)

We characterize and remedy a failure mode that may arise from multi-scale dynamics with scale imbalances during training of deep neural networks, such as Physics Informed Neural Networks (PINNs). PINNs are popular machine-learning templates that allow for seamless integration of physical equation models with data. Their training amounts to solving an optimization problem over a weighted sum of data-fidelity and equation-fidelity objectives. Conflicts between objectives can arise from scale imbalances, heteroscedasticity in the data, stiffness of the physical equation, or from catastrophic interference during sequential training. We explain the training pathology arising from this and propose a simple yet effective inverse-Dirichlet weighting strategy to alleviate the issue. We compare with Sobolev training of neural networks, providing the baseline of analytically boldsymbolε-optimal training. We demonstrate the effectiveness of inverse-Dirichlet weighting in various applications, including a multi-scale model of active turbulence, where we show orders of magnitude improvement in accuracy and convergence over conventional PINN training. For inverse modeling using sequential training, we find that inverse-Dirichlet weighting protects a PINN against catastrophic forgetting.