mlsanityData to 14 Oct 2026

Learnable Activation Functions in Physics-Informed Neural Networks for Solving Partial Differential Equations

arXiv 2411.15111 ↗First posted 22 Nov 2024cs.NE cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

Physics-Informed Neural Networks (PINNs) have emerged as a promising approach for solving Partial Differential Equations (PDEs). However, they face challenges related to spectral bias (the tendency to learn low-frequency components while struggling with high-frequency features) and unstable convergence dynamics (mainly stemming from the multi-objective nature of the PINN loss function). These limitations impact their accuracy for problems involving rapid oscillations, sharp gradients, and complex boundary behaviors. We systematically investigate learnable activation functions as a solution to these challenges, comparing Multilayer Perceptrons (MLPs) using fixed and learnable activation functions against Kolmogorov-Arnold Networks (KANs) that employ learnable basis functions. Our evaluation spans diverse PDE types, including linear and non-linear wave problems, mixed-physics systems, and fluid dynamics. Using empirical Neural Tangent Kernel (NTK) analysis and Hessian eigenvalue decomposition, we assess spectral bias and convergence stability of the models. Our results reveal a trade-off between expressivity and training convergence stability. While learnable activation functions work well in simpler architectures, they encounter scalability issues in complex networks due to the higher functional dimensionality. Counterintuitively, we find that low spectral bias alone does not guarantee better accuracy, as functions with broader NTK eigenvalue spectra may exhibit convergence instability. We demonstrate that activation function selection remains inherently problem-specific, with different bases showing distinct advantages for particular PDE characteristics. We believe these insights will help in the design of more robust neural PDE solvers.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 50 of its 79 references that are in the map. First posted on arXiv 22 Nov 2024.

Builds on

50 of 79 references.

References · 50

  1. 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,908
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  3. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,200
  4. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  5. 2018On the Spectral Bias of Neural Networkscited by 1,341
  6. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  7. 2024KAN: Kolmogorov-Arnold Networkscited by 649
  8. 2020When and why PINNs fail to train: A neural tangent kernel perspectivecited by 634
  9. 2022Scientific Machine Learning through Physics-Informed Neural Networks: Where we are and What's nextcited by 608
  10. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  11. 2022PDEBENCH: An Extensive Benchmark for Scientific Machine Learningcited by 413
  12. 2019Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networkscited by 368
38 more
  1. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 292
  2. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 292
  3. 2022A comprehensive study of non-adaptive and residual-based adaptive sampling for physics-informed neural networkscited by 236
  4. 2020Self-Adaptive Physics-Informed Neural Networks using a Soft Attention Mechanismcited by 233
  5. 2021Finite Basis Physics-Informed Neural Networks (FBPINNs): a scalable domain decomposition approach for solving differential equationscited by 145
  6. 2024Kolmogorov-Arnold Networks are Radial Basis Function Networkscited by 128
  7. 2024Wav-KAN: Wavelet Kolmogorov-Arnold Networkscited by 122
  8. 2024Chebyshev Polynomial-Based Kolmogorov-Arnold Networks: An Efficient Architecture for Nonlinear Function Approximationcited by 118
  9. 2024Challenges in Training PINNs: A Loss Landscape Perspectivecited by 113
  10. 2024A comprehensive and FAIR comparison between MLP and KAN representations for differential equations and operator networkscited by 104
  11. 2020A survey on modern trainable activation functionscited by 89
  12. 2021Multi-Objective Loss Balancing for Physics-Informed Deep Learningcited by 87
  13. 2023Residual-based attention in physics-informed neural networkscited by 72
  14. 2021DAS-PINNs: A deep adaptive sampling method for solving high-dimensional partial differential equationscited by 66
  15. 2024KAN-ODEs: Kolmogorov-Arnold Network Ordinary Differential Equations for Learning Dynamical Systems and Hidden Physicscited by 59
  16. 2022On the Maximum Hessian Eigenvalue and Generalizationcited by 57
  17. 2024BSRBF-KAN: A combination of B-splines and Radial Basis Functions in Kolmogorov-Arnold Networkscited by 46
  18. 2024Enhancing Graph Collaborative Filtering with FourierKAN Feature Transformationcited by 42
  19. 2019SoftAdapt: Techniques for Adaptive Loss Weighting of Neural Networks with Multi-Part Loss Functionscited by 40
  20. 2024Adaptive Training of Grid-Dependent Physics-Informed Kolmogorov-Arnold Networkscited by 38
  21. 2024On the expressiveness and spectral bias of KANscited by 37
  22. 2024Unveiling the Power of Wavelets: A Wavelet-based Kolmogorov-Arnold Network for Hyperspectral Image Classificationcited by 34
  23. 2021Hessian Eigenspectra of More Realistic Nonlinear Modelscited by 33
  24. 2022A mixed formulation for physics-informed neural networks as a potential solver for engineering problems in heterogeneous domains: comparison with finite element methodcited by 33
  25. 2022On the Activation Function Dependence of the Spectral Bias of Neural Networkscited by 28
  26. 2019Multi-scale Deep Neural Networks for Solving High Dimensional PDEscited by 25
  27. 2023Efficient physics-informed neural networks using hash encodingcited by 21
  28. 2019Negative eigenvalues of the Hessian in deep neural networkscited by 20
  29. 2021Reproducing Activation Function for Deep Learningcited by 19
  30. 2022The Spectral Bias of Polynomial Neural Networkscited by 19
  31. 2023A practical PINN framework for multi-scale problems with multi-magnitude loss termscited by 15
  32. 2023Learning Specialized Activation Functions for Physics-informed Neural Networkscited by 14
  33. 2018Adaptive Blending Units: Trainable Activation Functions for Deep Neural Networkscited by 13
  34. 2024Data-Driven Physics-Informed Neural Networks: A Digital Twin Perspectivecited by 12
  35. 2024Exact Enforcement of Temporal Continuity in Sequential Physics-Informed Neural Networkscited by 9
  36. 2024Binary structured physics-informed neural networks for solving equations with rapidly changing solutionscited by 8
  37. 2022Extrapolation and Spectral Bias of Neural Nets with Hadamard Product: a Polynomial Net Studycited by 7
  38. 2024Comparing Spectral Bias and Robustness For Two-Layer Neural Networks: SGD vs Adaptive Random Fourier Featurescited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 4.

7 papers.

  1. 2025A Practitioner's Guide to Kolmogorov-Arnold Networkscited by 18
  2. 2025QCPINN: Quantum-Classical Physics-Informed Neural Networks for Solving PDEscited by 9
  3. 2026Making Gaussian Kolmogorov-Arnold Networks Reliable and Accuratecited by 3
  4. 2025Learning Fluid-Structure Interaction with Physics-Informed Machine Learning and Immersed Boundary Methodscited by 0
  5. 2026High-Fidelity Modeling of Stochastic Chemical Dynamics on Complex Manifolds: A Multi-Scale SIREN-PINN Framework for the Curvature-Perturbed Ginzburg-Landau Equationcited by 0
  6. 2026A Family of Adaptive Activation Functions for Mitigating Failure Modes in Physics-Informed Neural Networkscited by 0
  7. 2026Residual Attention Physics-Informed Neural Networks for Robust Multiphysics Simulation of Steady-State Electrothermal Energy Systemscited by 0