# Gradient-Update Mismatch: Rethinking Conflict-Free Training of Physics-Informed Neural Networks

arXiv: https://arxiv.org/abs/2609.01558 · HTML: https://mlsanity.com/paper/2609.01558/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 31 references that are in the map.
- First posted on arXiv 1 Sep 2026.

## Builds on (most cited first, 21 of 21)

- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks](https://mlsanity.com/paper/1711.02257/) (2017)
- [Multi-Task Learning as Multi-Objective Optimization](https://mlsanity.com/paper/1810.04650/) (2018)
- [Characterizing possible failure modes in physics-informed neural networks](https://mlsanity.com/paper/2109.01050/) (2021)
- [When and why PINNs fail to train: A neural tangent kernel perspective](https://mlsanity.com/paper/2007.14527/) (2020)
- [Conflict-Averse Gradient Descent for Multi-task Learning](https://mlsanity.com/paper/2110.14048/) (2021)
- [Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training](https://mlsanity.com/paper/2305.14342/) (2023)
- [ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learning](https://mlsanity.com/paper/2006.00719/) (2020)
- [Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout](https://mlsanity.com/paper/2010.06808/) (2020)
- [Multi-Objective Loss Balancing for Physics-Informed Deep Learning](https://mlsanity.com/paper/2110.09813/) (2021)
- [PINNacle: A Comprehensive Benchmark of Physics-Informed Neural Networks for Solving PDEs](https://mlsanity.com/paper/2306.08827/) (2023)
- [Independent Component Alignment for Multi-Task Learning](https://mlsanity.com/paper/2305.19000/) (2023)
- [Gradient Alignment in Physics-informed Neural Networks: A Second-Order Optimization Perspective](https://mlsanity.com/paper/2502.00604/) (2025)
- [A Dual-Dimer Method for Training Physics-Constrained Neural Networks with Minimax Architecture](https://mlsanity.com/paper/2005.00615/) (2020)
- [ConFIG: Towards Conflict-free Training of Physics Informed Neural Networks](https://mlsanity.com/paper/2408.11104/) (2024)
- [Physics vs Distributions: Pareto Optimal Flow Matching with Physics Constraints](https://mlsanity.com/paper/2506.08604/) (2025)
- [Jacobian Descent for Multi-Objective Optimization](https://mlsanity.com/paper/2406.16232/) (2024)
- [An Overview on Machine Learning Methods for Partial Differential Equations: from Physics Informed Neural Networks to Deep Operator Learning](https://mlsanity.com/paper/2408.13222/) (2024)

## Built on by (most cited first, 1 of 1)

- [ORACLE: Optimizer-Relative Alignment for Constrained LEarning](https://mlsanity.com/paper/2610.09040/) (2026)

## Abstract (arXiv)

Training Physics-Informed Neural Networks (PINNs) requires jointly optimizing physics residual and initial/boundary condition loss terms, which often induce conflicting gradients. Gradient surgery methods mitigate this issue by constructing directions from loss-specific gradients to reduce conflict before optimizer transformation. However, even when the constructed direction is conflict-free, this property may not be preserved after optimizer transformation. Let at denote the direction constructed by gradient surgery, ut the optimizer proposal, and Ct the conflict-free cone induced by the loss-specific gradients. We show that modern optimizers can transform at through mechanisms such as historical state, adaptive scaling, preconditioning, or decoupled weight decay, so at ∈ Ct does not generally imply ut ∈ Ct. We refer to this optimizer-induced discrepancy in conflict-freeness between at and ut as Gradient-Update Mismatch (GUM). Accordingly, we propose Gradient-Update Alignment (GUA), which projects ut onto Ct to obtain the aligned update pt and applies pt to the parameters. When the optimizer maintains internal state, GUA further adjusts this state toward targets reconstructed from the applied update. We conduct extensive experiments and find that GUM is widespread across momentum, adaptive, and curvature-based optimizers, with conflict rates reaching up to 86.3%. Across all PINN settings, GUA achieves conflict-free applied updates and consistently improves various gradient surgery methods, reducing the relative L2 error by up to 98.2% in individual settings. Data and code are available at https://github.com/JingXiao10/GUA.