mlsanityData to 7 Oct 2026

PINNsFormer: A Transformer-Based Framework For Physics-Informed Neural Networks

arXiv 2307.11833 ↗First posted 21 Jul 2023cs.CE cs.LGcited by 62

Not on the map. Why?

Abstract (arXiv)

Physics-Informed Neural Networks (PINNs) have emerged as a promising deep learning framework for approximating numerical solutions to partial differential equations (PDEs). However, conventional PINNs, relying on multilayer perceptrons (MLP), neglect the crucial temporal dependencies inherent in practical physics systems and thus fail to propagate the initial condition constraints globally and accurately capture the true solutions under various scenarios. In this paper, we introduce a novel Transformer-based framework, termed PINNsFormer, designed to address this limitation. PINNsFormer can accurately approximate PDE solutions by utilizing multi-head attention mechanisms to capture temporal dependencies. PINNsFormer transforms point-wise inputs into pseudo sequences and replaces point-wise PINNs loss with a sequential loss. Additionally, it incorporates a novel activation function, Wavelet, which anticipates Fourier decomposition through deep neural networks. Empirical results demonstrate that PINNsFormer achieves superior generalization ability and accuracy across various scenarios, including PINNs failure modes and high-dimensional PDEs. Moreover, PINNsFormer offers flexibility in integrating existing learning schemes for PINNs, further enhancing its performance.

Key facts. Cited by 62 papers in the map (data through 7 Oct 2026). Builds on 28 of its 49 references that are in the map. First posted on arXiv 21 Jul 2023.

Builds on

28 of 49 references.

References · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  7. 2018Neural Ordinary Differential Equationscited by 3,733
  8. 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,806
  9. 2020Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingcited by 2,141
  10. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  11. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  12. 2021Characterizing possible failure modes in physics-informed neural networkscited by 632
16 more
  1. 2020When and why PINNs fail to train: A neural tangent kernel perspectivecited by 629
  2. 2017Convolutional Sequence to Sequence Learningcited by 550
  3. 2017Solving high-dimensional partial differential equations using deep learningcited by 549
  4. 2017Physics Informed Deep Learning (Part I): Data-driven Solutions of Nonlinear Partial Differential Equationscited by 362
  5. 2022How Do Vision Transformers Work?cited by 305
  6. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 291
  7. 2021Physics-informed neural networks with hard constraints for inverse designcited by 277
  8. 2019Physics-Constrained Deep Learning for High-dimensional Surrogate Modeling and Uncertainty Quantification without Labeled Datacited by 261
  9. 2018Deep Hidden Physics Models: Deep Learning of Nonlinear Partial Differential Equationscited by 233
  10. 2020Self-Adaptive Physics-Informed Neural Networks using a Soft Attention Mechanismcited by 231
  11. 2020Physics-informed learning of governing equations from scarce datacited by 167
  12. 2018Physics-Informed Generative Adversarial Networks for Stochastic Differential Equationscited by 133
  13. 2021Learning in Sinusoidal Spaces with Physics-Informed Neural Networkscited by 61
  14. 2021Quadratic Residual Networks: A New Class of Neural Networks for Solving Forward and Inverse Problems in Physics Involving PDEscited by 42
  15. 2022Lagrangian PINNs: A causality-conforming solution to failure modes of physics-informed neural networkscited by 26
  16. 2022Auto-PINN: Understanding and Optimizing Physics-Informed Neural Architecturecited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 26; 2026: 21.

62 papers.

Sort:

By area:

  1. 2024From PINNs to PIKANs: Recent Advances in Physics-Informed Machine Learningcited by 73
  2. 2024Recent Advances on Machine Learning for Computational Fluid Dynamics: A Surveycited by 27
  3. 2024RoPINN: Region Optimized Physics-Informed Neural Networkscited by 26
  4. 2024Dual Cone Gradient Descent for Training Physics-Informed Neural Networkscited by 21
  5. 2024Zero-Shot Uncertainty Quantification using Diffusion Probabilistic Modelscited by 14
  6. 2025Always Skip Attentioncited by 12
  7. 2025FP64 is All You Need: Rethinking Failure Modes in Physics-Informed Neural Networkscited by 11
  8. 2025TRACE: Learning 3D Gaussian Physical Dynamics from Multi-view Videoscited by 11
  9. 2025Sub-Sequential Physics-Informed Learning with State Space Modelcited by 10
  10. 2024Rethinking Attention: Polynomial Alternatives to Softmax in Transformerscited by 8
  11. 2025Enhancing Physics-Informed Neural Networks Through Feature Engineeringcited by 8
  12. 2024Structure-Preserving Physics-Informed Neural Networks With Energy or Lyapunov Structurecited by 6
38 more
  1. 2024Physics-Informed Neural Networks for Dynamic Process Operations with Limited Physical Knowledge and Datacited by 6
  2. 2025ProPINN: Demystifying Propagation Failures in Physics-Informed Neural Networkscited by 6
  3. 2025Complex Physics-Informed Neural Networkcited by 6
  4. 2025Convolution-weighting method for the physics-informed neural network: A Primal-Dual Optimization Perspectivecited by 6
  5. 2024TGPT-PINN: Nonlinear model reduction with transformed GPT-PINNscited by 5
  6. 2025Physics-informed Temporal Alignment for Auto-regressive PDE Foundation Modelscited by 5
  7. 2024Cauchy activation function and XNetcited by 4
  8. 2025FD-Bench: A Modular and Fair Benchmark for Data-driven Fluid Simulationcited by 4
  9. 2025Ultra Memory-Efficient On-FPGA Training of Transformers via Tensor-Compressed Optimizationcited by 3
  10. 2025Neuro-Spectral Architectures for Causal Physics-Informed Networkscited by 3
  11. 2025F-Adapter: Frequency-Adaptive Parameter-Efficient Fine-Tuning in Scientific Machine Learningcited by 3
  12. 2024Feature Mapping in Physics-Informed Neural Networks (PINNs)cited by 2
  13. 2024SetPINNs: Set-based Physics-informed Neural Networkscited by 2
  14. 2025Hybrid Two-Stage Reconstruction of Multiscale Subsurface Flow with Physics-informed Residual Connected Neural Operatorcited by 2
  15. 2025Beyond Accuracy: EcoL2 Metric for Sustainable Neural PDE Solverscited by 2
  16. 2025Hierarchical-embedding autoencoder with a predictor (HEAP) as efficient architecture for learning long-term evolution of complex multi-scale physical systemscited by 2
  17. 2025MMET: A Multi-Input and Multi-Scale Transformer for Efficient PDEs Solvingcited by 2
  18. 2025Revisiting Heat Flux Analysis of Tungsten Monoblock Divertor on EAST using Physics-Informed Neural Networkcited by 2
  19. 2025Iterative Training of Physics-Informed Neural Networks with Fourier-enhanced Featurescited by 2
  20. 2026A Physics-Informed Fourier-Wavelet Transformer for Multiscale Computational Fluid Dynamics Surrogate Modelingcited by 2
  21. 2025Unraveling particle dark matter with Physics-Informed Neural Networkscited by 1
  22. 2025Enhancing Stability of Physics-Informed Neural Network Training Through Saddle-Point Reformulationcited by 1
  23. 2026Hybrid operator learning of wave scattering maps in high-contrast mediacited by 1
  24. 2026In Transformer We Trust? A Perspective on Transformer Architecture Failure Modescited by 1
  25. 2026Per-Loss Adapters for Gradient Conflict in Physics-Informed Neural Networkscited by 1
  26. 2024Architectural Strategies for the optimization of Physics-Informed Neural Networkscited by 0
  27. 2024Enhancing Multiscale Simulations with Constitutive Relations-Aware Deep Operator Networkscited by 0
  28. 2024SEA: State-Exchange Attention for High-Fidelity Physics Based Transformerscited by 0
  29. 2025Enhancing Neural Function Approximation: The XNet Outperforming KANcited by 0
  30. 2025Physics Consistency and Latent Dynamics in Spatiotemporal Physics Field Generationcited by 0
  31. 2025Mollifier Layers: Enabling Efficient High-Order Derivatives in Inverse PDE Learningcited by 0
  32. 2025Learning Mappings in Mesh-based Simulationscited by 0
  33. 2025Physics-Informed High-order Graph Dynamics Identification Learning for Predicting Complex Networks Long-term Dynamicscited by 0
  34. 2026Solving stiff dark matter equations via Jacobian Normalization with Physics-Informed Neural Networkscited by 0
  35. 2026A Family of Adaptive Activation Functions for Mitigating Failure Modes in Physics-Informed Neural Networkscited by 0
  36. 2026Deep Wave Network for Modeling Multi-Scale Physical Dynamicscited by 0
  37. 2026Chebyshev Center-Based Direction Selection for Multi-Objective Optimization and Training PINNscited by 0
  38. 2026Mitigating Gradient Pathology in PINNs through Aligned Constraintcited by 0

50 most cited of 62 All 62 (JSON)