mlsanityData to 14 Oct 2026

Momentum Diminishes the Effect of Spectral Bias in Physics-Informed Neural Networks

arXiv 2206.14862 ↗First posted 29 Jun 2022cs.LG cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

Physics-informed neural network (PINN) algorithms have shown promising results in solving a wide range of problems involving partial differential equations (PDEs). However, they often fail to converge to desirable solutions when the target function contains high-frequency features, due to a phenomenon known as spectral bias. In the present work, we exploit neural tangent kernels (NTKs) to investigate the training dynamics of PINNs evolving under stochastic gradient descent with momentum (SGDM). This demonstrates SGDM significantly reduces the effect of spectral bias. We have also examined why training a model via the Adam optimizer can accelerate the convergence while reducing the spectral bias. Moreover, our numerical experiments have confirmed that wide-enough networks using SGDM still converge to desirable solutions, even in the presence of high-frequency features. In fact, we show that the width of a network plays a critical role in convergence.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 18 of its 26 references that are in the map. First posted on arXiv 29 Jun 2022.

Builds on

18 of 26 references.

References · 18

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  3. 2018On the Spectral Bias of Neural Networkscited by 1,341
  4. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  5. 2017DGM: A deep learning algorithm for solving partial differential equationscited by 678
  6. 2021Characterizing possible failure modes in physics-informed neural networkscited by 643
  7. 2020When and why PINNs fail to train: A neural tangent kernel perspectivecited by 634
  8. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  9. 2021Physics-informed neural networks (PINNs) for fluid mechanics: A reviewcited by 473
  10. 2020Loss landscapes and optimization in over-parameterized non-linear systems and neural networkscited by 233
  11. 2019Towards Understanding the Spectral Bias of Deep Learningcited by 174
  12. 2019Variational Physics-Informed Neural Networks For Solving Partial Differential Equationscited by 148
6 more
  1. 2020Frequency Bias in Neural Networks for Input of Non-Uniform Densitycited by 140
  2. 2017Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradientscited by 139
  3. 2019The Convergence Rate of Neural Networks for Learned Functions of Different Frequenciescited by 135
  4. 2020On the linearity of large non-linear models: when and why the tangent kernel is constantcited by 129
  5. 2018Deep UQ: Learning deep neural network surrogate models for high dimensional uncertainty quantificationcited by 81
  6. 2019Machine learning in cardiovascular flows modeling: Predicting arterial blood pressure from non-invasive 4D flow MRI data using physics-informed neural networkscited by 76

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 2; 2025: 1.

4 papers.

  1. 2024Towards General Neural Surrogate Solvers with Specialized Neural Acceleratorscited by 9
  2. 2023A Block-Coordinate Approach of Multi-level Optimization with an Application to Physics-Informed Neural Networkscited by 7
  3. 2024Physics-Informed Neural Networks: Minimizing Residual Loss with Wide Networks and Effective Activationscited by 4
  4. 2025xLSTM-PINN: Memory-Gated Spectral Remodeling for Physics-Informed Learningcited by 0