# Dual Cone Gradient Descent for Training Physics-Informed Neural Networks

arXiv: https://arxiv.org/abs/2409.18426 · HTML: https://mlsanity.com/paper/2409.18426/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 48 references that are in the map.
- First posted on arXiv 27 Sep 2024.

## Builds on (most cited first, 29 of 29)

- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [Multi-Task Learning as Multi-Objective Optimization](https://mlsanity.com/paper/1810.04650/) (2018)
- [Characterizing possible failure modes in physics-informed neural networks](https://mlsanity.com/paper/2109.01050/) (2021)
- [When and why PINNs fail to train: A neural tangent kernel perspective](https://mlsanity.com/paper/2007.14527/) (2020)
- [Conflict-Averse Gradient Descent for Multi-task Learning](https://mlsanity.com/paper/2110.14048/) (2021)
- [hp-VPINNs: Variational Physics-Informed Neural Networks With Domain Decomposition](https://mlsanity.com/paper/2003.05385/) (2020)
- [A comprehensive study of non-adaptive and residual-based adaptive sampling for physics-informed neural networks](https://mlsanity.com/paper/2207.10289/) (2022)
- [Gradient-enhanced physics-informed neural networks for forward and inverse PDE problems](https://mlsanity.com/paper/2111.02801/) (2021)
- [Multi-Task Learning as a Bargaining Game](https://mlsanity.com/paper/2202.01017/) (2022)
- [Respecting causality is all you need for training physics-informed neural networks](https://mlsanity.com/paper/2203.07404/) (2022)
- [PINNacle: A Comprehensive Benchmark of Physics-Informed Neural Networks for Solving PDEs](https://mlsanity.com/paper/2306.08827/) (2023)
- [Mitigating Propagation Failures in Physics-informed Neural Networks using Retain-Resample-Release (R3) Sampling](https://mlsanity.com/paper/2207.02338/) (2022)
- [Independent Component Alignment for Multi-Task Learning](https://mlsanity.com/paper/2305.19000/) (2023)
- [Separable Physics-Informed Neural Networks](https://mlsanity.com/paper/2306.15969/) (2023)
- [Learning in Modal Space: Solving Time-Dependent Stochastic PDEs Using Physics-Informed Neural Networks](https://mlsanity.com/paper/1905.01205/) (2019)
- [PINNsFormer: A Transformer-Based Framework For Physics-Informed Neural Networks](https://mlsanity.com/paper/2307.11833/) (2023)
- [Learning in Sinusoidal Spaces with Physics-Informed Neural Networks](https://mlsanity.com/paper/2109.09338/) (2021)
- [DPM: A Novel Training Method for Physics-Informed Neural Networks in Extrapolation](https://mlsanity.com/paper/2012.02681/) (2020)
- [Hypernetwork-based Meta-Learning for Low-Rank Physics-Informed Neural Networks](https://mlsanity.com/paper/2310.09528/) (2023)
- [Enhanced Physics-Informed Neural Networks with Augmented Lagrangian Relaxation Method (AL-PINNs)](https://mlsanity.com/paper/2205.01059/) (2022)
- [Position: Optimization in SciML Should Employ the Function Space Geometry](https://mlsanity.com/paper/2402.07318/) (2024)
- [Fully probabilistic deep models for forward and inverse problems in parametric PDEs](https://mlsanity.com/paper/2208.04856/) (2022)
- [Sliced Denoising: A Physics-Informed Molecular Pre-Training Method](https://mlsanity.com/paper/2311.02124/) (2023)
- [MultiAdam: Parameter-wise Scale-invariant Optimizer for Multiscale Training of Physics-informed Neural Networks](https://mlsanity.com/paper/2306.02816/) (2023)
- [Training multi-objective/multi-task collocation physics-informed neural network with student/teachers transfer learnings](https://mlsanity.com/paper/2107.11496/) (2021)
- [Understanding and Mitigating Extrapolation Failures in Physics-Informed Neural Networks](https://mlsanity.com/paper/2306.09478/) (2023)
- [Random Grid Neural Processes for Parametric Partial Differential Equations](https://mlsanity.com/paper/2301.11040/) (2023)
- [Physics Informed RNN-DCT Networks for Time-Dependent Partial Differential Equations](https://mlsanity.com/paper/2202.12358/) (2022)
- [Hierarchical Learning to Solve Partial Differential Equations Using Physics-Informed Neural Networks](https://mlsanity.com/paper/2112.01254/) (2021)

## Built on by (most cited first, 21 of 21)

- [Gradient-Based Multi-Objective Deep Learning: Algorithms, Theories, Applications, and Beyond](https://mlsanity.com/paper/2501.10945/) (2025)
- [Curvature-Aware Optimization for High-Accuracy Physics-Informed Neural Networks](https://mlsanity.com/paper/2604.05230/) (2026)
- [Learning PDE Solvers with Physics and Data: A Unifying View of Physics-Informed Neural Networks and Neural Operators](https://mlsanity.com/paper/2601.14517/) (2026)
- [Neural Field Thermal Tomography: A Differentiable Physics Framework for Non-Destructive Evaluation](https://mlsanity.com/paper/2603.11045/) (2026)
- [Tackling Failure Modes of PINNs and PIKANs Using Conflict-Free Gradients](https://mlsanity.com/paper/2609.14841/) (2026)
- [Enhancing Stability of Physics-Informed Neural Network Training Through Saddle-Point Reformulation](https://mlsanity.com/paper/2507.16008/) (2025)
- [naPINN: Noise-Adaptive Physics-Informed Neural Networks for Recovering Physics from Corrupted Measurement](https://mlsanity.com/paper/2602.02547/) (2026)
- [Per-Loss Adapters for Gradient Conflict in Physics-Informed Neural Networks](https://mlsanity.com/paper/2605.10136/) (2026)
- [CRAFT: Conflict-Resolved Aggregation for Federated Training](https://mlsanity.com/paper/2605.21317/) (2026)
- [Reconstructing Physics-Informed Machine Learning for Traffic Flow Modeling: a Multi-Gradient Descent and Pareto Learning Approach](https://mlsanity.com/paper/2505.13241/) (2025)
- [Taylor-Model Physics-Informed Neural Networks (PINNs) for Ordinary Differential Equations](https://mlsanity.com/paper/2507.03860/) (2025)
- [Breaking the Precision Ceiling in Physics-Informed Neural Networks: A Hybrid Fourier-Neural Architecture for Ultra-High Accuracy](https://mlsanity.com/paper/2507.20929/) (2025)
- [Controllable Machine Unlearning via Gradient Pivoting](https://mlsanity.com/paper/2510.19226/) (2025)
- [Synergizing Kolmogorov-Arnold Networks with Dynamic Adaptive Weighting for High-Frequency and Multi-Scale PDE Solutions](https://mlsanity.com/paper/2512.22283/) (2025)
- [Chebyshev Center-Based Direction Selection for Multi-Objective Optimization and Training PINNs](https://mlsanity.com/paper/2605.09975/) (2026)
- [MetaColloc: Optimization-Free PDE Solving via Meta-Learned Basis Functions](https://mlsanity.com/paper/2605.12368/) (2026)
- [Mitigating Gradient Pathology in PINNs through Aligned Constraint](https://mlsanity.com/paper/2605.25001/) (2026)
- [A Unified Framework for Gradient Aggregation in Multi-Objective Optimization](https://mlsanity.com/paper/2605.30452/) (2026)
- [PINNs Failure Modes are Overfitting](https://mlsanity.com/paper/2605.30910/) (2026)
- [Modularity-Free Conflict-Averse Training for Generalized PINNs](https://mlsanity.com/paper/2606.20156/) (2026)
- [Physics and Data Driven Transformer-Mamba Framework for Flow Field](https://mlsanity.com/paper/2609.29087/) (2026)

## Abstract (arXiv)

Physics-informed neural networks (PINNs) have emerged as a prominent approach for solving partial differential equations (PDEs) by minimizing a combined loss function that incorporates both boundary loss and PDE residual loss. Despite their remarkable empirical performance in various scientific computing tasks, PINNs often fail to generate reasonable solutions, and such pathological behaviors remain difficult to explain and resolve. In this paper, we identify that PINNs can be adversely trained when gradients of each loss function exhibit a significant imbalance in their magnitudes and present a negative inner product value. To address these issues, we propose a novel optimization framework, Dual Cone Gradient Descent (DCGD), which adjusts the direction of the updated gradient to ensure it falls within a dual cone region. This region is defined as a set of vectors where the inner products with both the gradients of the PDE residual loss and the boundary loss are non-negative. Theoretically, we analyze the convergence properties of DCGD algorithms in a non-convex setting. On a variety of benchmark equations, we demonstrate that DCGD outperforms other optimization algorithms in terms of various evaluation metrics. In particular, DCGD achieves superior predictive accuracy and enhances the stability of training for failure modes of PINNs and complex PDEs, compared to existing optimally tuned models. Moreover, DCGD can be further improved by combining it with popular strategies for PINNs, including learning rate annealing and the Neural Tangent Kernel (NTK).