# Learning in Sinusoidal Spaces with Physics-Informed Neural Networks

arXiv: https://arxiv.org/abs/2109.09338 · HTML: https://mlsanity.com/paper/2109.09338/

## Key facts

- Cited by 62 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 68 references that are in the map.
- First posted on arXiv 20 Sep 2021.

## Builds on (most cited first, 30 of 34)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [Automatic differentiation in machine learning: a survey](https://mlsanity.com/paper/1502.05767/) (2015)
- [DGM: A deep learning algorithm for solving partial differential equations](https://mlsanity.com/paper/1708.07469/) (2017)
- [When and why PINNs fail to train: A neural tangent kernel perspective](https://mlsanity.com/paper/2007.14527/) (2020)
- [DeepXDE: A deep learning library for solving differential equations](https://mlsanity.com/paper/1907.04502/) (2019)
- [The Deep Ritz method: A deep learning-based numerical algorithm for solving variational problems](https://mlsanity.com/paper/1710.00211/) (2017)
- [Learning the solution operator of parametric partial differential equations with physics-informed DeepOnets](https://mlsanity.com/paper/2103.10974/) (2021)
- [On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks](https://mlsanity.com/paper/2012.10047/) (2020)
- [Deep Hidden Physics Models: Deep Learning of Nonlinear Partial Differential Equations](https://mlsanity.com/paper/1801.06637/) (2018)
- [Self-Adaptive Physics-Informed Neural Networks using a Soft Attention Mechanism](https://mlsanity.com/paper/2009.04544/) (2020)
- [Informed Machine Learning -- A Taxonomy and Survey of Integrating Knowledge into Learning Systems](https://mlsanity.com/paper/1903.12394/) (2019)
- [Training behavior of deep neural network in frequency domain](https://mlsanity.com/paper/1807.01251/) (2018)
- [PhyGeoNet: Physics-Informed Geometry-Adaptive Convolutional Neural Networks for Solving Parameterized Steady-State PDEs on Irregular Domain](https://mlsanity.com/paper/2004.13145/) (2020)
- [Theory-guided Data Science: A New Paradigm for Scientific Discovery from Data](https://mlsanity.com/paper/1612.08544/) (2016)
- [Understanding and mitigating gradient pathologies in physics-informed neural networks](https://mlsanity.com/paper/2001.04536/) (2020)
- [Physics-guided Neural Networks (PGNN): An Application in Lake Temperature Modeling](https://mlsanity.com/paper/1710.11431/) (2017)
- [A unified deep artificial neural network approach to partial differential equations in complex geometries](https://mlsanity.com/paper/1711.06464/) (2017)
- [Transfer learning enhanced physics informed neural network for phase-field modeling of fracture](https://mlsanity.com/paper/1907.02531/) (2019)
- [Deep Learning of Vortex Induced Vibrations](https://mlsanity.com/paper/1808.08952/) (2018)
- [Physics-informed neural network for ultrasound nondestructive quantification of surface breaking cracks](https://mlsanity.com/paper/2005.03596/) (2020)
- [The Old and the New: Can Physics-Informed Deep-Learning Replace Traditional Linear Solvers?](https://mlsanity.com/paper/2103.09655/) (2021)
- [NVIDIA SimNet^{TM}: an AI-accelerated multi-physics simulation framework](https://mlsanity.com/paper/2012.07938/) (2020)
- [A Physics-Informed Machine Learning Approach for Solving Heat Transfer Equation in Advanced Manufacturing and Engineering Applications](https://mlsanity.com/paper/2010.02011/) (2020)
- [D3M: A deep domain decomposition method for partial differential equations](https://mlsanity.com/paper/1909.12236/) (2019)
- [Spline-PINN: Approaching PDEs without Data using Fast, Physics-Informed Hermite-Spline CNNs](https://mlsanity.com/paper/2109.07143/) (2021)
- [A Physics-Guided Neural Network Framework for Elastic Plates: Comparison of Governing Equations-Based and Energy-Based Approaches](https://mlsanity.com/paper/2010.06050/) (2020)

## Built on by (most cited first, 30 of 62)

- [Scientific Machine Learning through Physics-Informed Neural Networks: Where we are and What's next](https://mlsanity.com/paper/2201.05624/) (2022)
- [CAN-PINN: A Fast Physics-Informed Neural Network Based on Coupled-Automatic-Numerical Differentiation Method](https://mlsanity.com/paper/2110.15832/) (2021)
- [Physics-Informed Machine Learning: A Survey on Problems, Methods and Applications](https://mlsanity.com/paper/2211.08064/) (2022)
- [Mitigating Propagation Failures in Physics-informed Neural Networks using Retain-Resample-Release (R3) Sampling](https://mlsanity.com/paper/2207.02338/) (2022)
- [PINNsFormer: A Transformer-Based Framework For Physics-Informed Neural Networks](https://mlsanity.com/paper/2307.11833/) (2023)
- [ConFIG: Towards Conflict-free Training of Physics Informed Neural Networks](https://mlsanity.com/paper/2408.11104/) (2024)
- [Enhanced Physics-Informed Neural Networks with Augmented Lagrangian Relaxation Method (AL-PINNs)](https://mlsanity.com/paper/2205.01059/) (2022)
- [RoPINN: Region Optimized Physics-Informed Neural Networks](https://mlsanity.com/paper/2405.14369/) (2024)
- [On the Role of Fixed Points of Dynamical Systems in Training Physics-Informed Neural Networks](https://mlsanity.com/paper/2203.13648/) (2022)
- [Dual Cone Gradient Descent for Training Physics-Informed Neural Networks](https://mlsanity.com/paper/2409.18426/) (2024)
- [Improved Training of Physics-Informed Neural Networks with Model Ensembles](https://mlsanity.com/paper/2204.05108/) (2022)
- [Noise-aware Physics-informed Machine Learning for Robust PDE Discovery](https://mlsanity.com/paper/2206.12901/) (2022)
- [Learning Specialized Activation Functions for Physics-informed Neural Networks](https://mlsanity.com/paper/2308.04073/) (2023)
- [Data-Driven Physics-Informed Neural Networks: A Digital Twin Perspective](https://mlsanity.com/paper/2401.08667/) (2024)
- [Evolutionary Optimization of Physics-Informed Neural Networks: Evo-PINN Frontiers and Opportunities](https://mlsanity.com/paper/2501.06572/) (2025)
- [Sub-Sequential Physics-Informed Learning with State Space Model](https://mlsanity.com/paper/2502.00318/) (2025)
- [Simple initialization and parametrization of sinusoidal networks via their kernel bandwidth](https://mlsanity.com/paper/2211.14503/) (2022)
- [ProPINN: Demystifying Propagation Failures in Physics-Informed Neural Networks](https://mlsanity.com/paper/2502.00803/) (2025)
- [Evolutionary Optimization of Physics-Informed Neural Networks: Advancing Generalizability by the Baldwin Effect](https://mlsanity.com/paper/2312.03243/) (2023)
- [RBF-PINN: Non-Fourier Positional Embedding in Physics-Informed Neural Networks](https://mlsanity.com/paper/2402.08367/) (2024)
- [Data-Guided Physics-Informed Neural Networks for Solving Inverse Problems in Partial Differential Equations](https://mlsanity.com/paper/2407.10836/) (2024)
- [Scale-PINN: Learning Efficient Physics-Informed Neural Networks Through Sequential Correction](https://mlsanity.com/paper/2602.19475/) (2026)
- [Neuroevolution of Physics-Informed Neural Nets: Benchmark Problems and Comparative Results](https://mlsanity.com/paper/2212.07624/) (2022)
- [LSA-PINN: Linear Boundary Connectivity Loss for Solving PDEs on Complex Geometry](https://mlsanity.com/paper/2302.01518/) (2023)
- [Physics-Informed Neural Networks: Minimizing Residual Loss with Wide Networks and Effective Activations](https://mlsanity.com/paper/2405.01680/) (2024)
- [Beyond Derivative Pathology of PINNs: Variable Splitting Strategy with Convergence Analysis](https://mlsanity.com/paper/2409.20383/) (2024)
- [Robustness of Physics-Informed Neural Networks to Noise in Sensor Data](https://mlsanity.com/paper/2211.12042/) (2022)
- [Decoder Decomposition for the Analysis of the Latent Space of Nonlinear Autoencoders With Wind-Tunnel Experimental Data](https://mlsanity.com/paper/2404.19660/) (2024)
- [Gabor-Enhanced Physics-Informed Neural Networks for Fast Simulations of Acoustic Wavefields](https://mlsanity.com/paper/2502.17134/) (2025)
- [iPINNER: An Iterative Physics-Informed Neural Network with Ensemble Kalman Filter](https://mlsanity.com/paper/2506.00731/) (2025)

## Abstract (arXiv)

A physics-informed neural network (PINN) uses physics-augmented loss functions, e.g., incorporating the residual term from governing partial differential equations (PDEs), to ensure its output is consistent with fundamental physics laws. However, it turns out to be difficult to train an accurate PINN model for many problems in practice. In this paper, we present a novel perspective of the merits of learning in sinusoidal spaces with PINNs. By analyzing behavior at model initialization, we first show that a PINN of increasing expressiveness induces an initial bias around flat output functions. Notably, this initial solution can be very close to satisfying many physics PDEs, i.e., falling into a local minimum of the PINN loss that only minimizes PDE residuals, while still being far from the true solution that jointly minimizes PDE residuals and the initial and/or boundary conditions. It is difficult for gradient descent optimization to escape from such a local minimum trap, often causing the training to stall. We then prove that the sinusoidal mapping of inputs, in an architecture we label as sf-PINN, is effective to increase input gradient variability, thus avoiding being trapped in such deceptive local minimum. The level of variability can be effectively modulated to match high-frequency patterns in the problem at hand. A key facet of this paper is the comprehensive empirical study that demonstrates the efficacy of learning in sinusoidal spaces with PINNs for a wide range of forward and inverse modelling problems spanning multiple physics domains.