mlsanityData to 7 Oct 2026

Physics Informed Token Transformer for Solving Partial Differential Equations

arXiv 2305.08757 ↗First posted 15 May 2023cs.LG physics.comp-phcited by 21

Not on the map. Why?

Abstract (arXiv)

Solving Partial Differential Equations (PDEs) is the core of many fields of science and engineering. While classical approaches are often prohibitively slow, machine learning models often fail to incorporate complete system information. Over the past few years, transformers have had a significant impact on the field of Artificial Intelligence and have seen increased usage in PDE applications. However, despite their success, transformers currently lack integration with physics and reasoning. This study aims to address this issue by introducing PITT: Physics Informed Token Transformer. The purpose of PITT is to incorporate the knowledge of physics by embedding partial differential equations (PDEs) into the learning process. PITT uses an equation tokenization method to learn an analytically-driven numerical update operator. By tokenizing PDEs and embedding partial derivatives, the transformer models become aware of the underlying knowledge behind physical processes. To demonstrate this, PITT is tested on challenging 1D and 2D PDE neural operator prediction tasks. The results show that PITT outperforms popular neural operator models and has the ability to extract physically relevant information from governing equations.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 25 of its 42 references that are in the map. First posted on arXiv 15 May 2023.

Builds on

25 of 42 references.

References · 25

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  6. 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,806
  7. 2019DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operatorscited by 1,965
  8. 2021Characterizing possible failure modes in physics-informed neural networkscited by 632
  9. 2021Neural Operator: Learning Maps Between Function Spacescited by 413
  10. 2022PDEBENCH: An Extensive Benchmark for Scientific Machine Learningcited by 396
  11. 2022Message Passing Neural PDE Solverscited by 368
  12. 2021Choose a Transformer: Fourier or Galerkincited by 333
13 more
  1. 2023GNOT: A General Neural Operator Transformer for Operator Learningcited by 321
  2. 2022Transformer for Partial Differential Equations' Operator Learningcited by 270
  3. 2019Deep Learning for Symbolic Mathematicscited by 266
  4. 2022A Physics-informed Diffusion Model for High-fidelity Flow Field Reconstructioncited by 150
  5. 2020Transformers for Modeling Physical Systemscited by 105
  6. 2022Learning Operators with Coupled Attentioncited by 88
  7. 2022Predicting Physics in Mesh-reduced Space with Temporal Attentioncited by 82
  8. 2021Meta-Learning Dynamics Forecasting Using Task Inferencecited by 37
  9. 2021Reinforcement Learning for Adaptive Mesh Refinementcited by 37
  10. 2021Deep Learning for Reduced Order Modelling and Efficient Temporal Evolution of Fluid Simulationscited by 36
  11. 2022Transformer Meets Boundary Value Inverse Problemscited by 22
  12. 2023Learning Controllable Adaptive Simulation for Multi-resolution Physicscited by 20
  13. 2022Deep Reinforcement Learning for Adaptive Mesh Refinementcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 10; 2025: 4; 2026: 6.

21 papers.

Sort:

By area:

  1. 2024Unisolver: PDE-Conditional Transformers Towards Universal Neural PDE Solverscited by 31
  2. 2024VICON: Vision In-Context Operator Networks for Multi-Physics Fluid Dynamics Predictioncited by 22
  3. 2024Strategies for Pretraining Neural Operatorscited by 14
  4. 2024Predicting Change, Not States: An Alternate Framework for Neural PDE Surrogatescited by 8
  5. 2025A Multimodal PDE Foundation Model for Prediction and Scientific Text Descriptionscited by 8
  6. 2024Explain Like I'm Five: Using LLMs to Improve PDE Surrogate Models with Textcited by 6
  7. 2024PICL: Physics Informed Contrastive Learning for Partial Differential Equationscited by 5
  8. 2024Latent Neural PDE Solver: a reduced-order modelling framework for partial differential equationscited by 5
  9. 2023Hyena Neural Operator for Partial Differential Equationscited by 4
  10. 2025Flow marching for a generative PDE foundation modelcited by 3
  11. 2026Latent Generative Solvers for Generalizable Long-Term Physics Simulationcited by 2
  12. 2026A Physics-Informed Fourier-Wavelet Transformer for Multiscale Computational Fluid Dynamics Surrogate Modelingcited by 2
9 more
  1. 2024Inpainting Computational Fluid Dynamics with Deep Learningcited by 1
  2. 2024Attention-Based Learning for Fluid State Interpolation and Editing in a Time-Continuous Frameworkcited by 1
  3. 2025Generalizing PDE Emulation with Equation-Aware Neural Operatorscited by 1
  4. 2026In Transformer We Trust? A Perspective on Transformer Architecture Failure Modescited by 1
  5. 2024SysCaps: Language Interfaces for Simulation Surrogates of Complex Systemscited by 0
  6. 2025FieldFormer: Locality-Aware Transformers for Spatio-Temporal Modeling on Sparse Sensor Networkscited by 0
  7. 2026LLT: Local Linear Transformer for PDE Operator Learningcited by 0
  8. 2026Large language models for partial differential equation workflowscited by 0
  9. 2026RD-JEPA: Predictive latent pretraining for few-trajectory transfer across reaction--diffusion equationscited by 0