# Physics Informed Token Transformer for Solving Partial Differential Equations

arXiv: https://arxiv.org/abs/2305.08757 · HTML: https://mlsanity.com/paper/2305.08757/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 42 references that are in the map.
- First posted on arXiv 15 May 2023.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [Characterizing possible failure modes in physics-informed neural networks](https://mlsanity.com/paper/2109.01050/) (2021)
- [Neural Operator: Learning Maps Between Function Spaces](https://mlsanity.com/paper/2108.08481/) (2021)
- [PDEBENCH: An Extensive Benchmark for Scientific Machine Learning](https://mlsanity.com/paper/2210.07182/) (2022)
- [Message Passing Neural PDE Solvers](https://mlsanity.com/paper/2202.03376/) (2022)
- [Choose a Transformer: Fourier or Galerkin](https://mlsanity.com/paper/2105.14995/) (2021)
- [GNOT: A General Neural Operator Transformer for Operator Learning](https://mlsanity.com/paper/2302.14376/) (2023)
- [Transformer for Partial Differential Equations' Operator Learning](https://mlsanity.com/paper/2205.13671/) (2022)
- [Deep Learning for Symbolic Mathematics](https://mlsanity.com/paper/1912.01412/) (2019)
- [A Physics-informed Diffusion Model for High-fidelity Flow Field Reconstruction](https://mlsanity.com/paper/2211.14680/) (2022)
- [Transformers for Modeling Physical Systems](https://mlsanity.com/paper/2010.03957/) (2020)
- [Learning Operators with Coupled Attention](https://mlsanity.com/paper/2201.01032/) (2022)
- [Predicting Physics in Mesh-reduced Space with Temporal Attention](https://mlsanity.com/paper/2201.09113/) (2022)
- [Reinforcement Learning for Adaptive Mesh Refinement](https://mlsanity.com/paper/2103.01342/) (2021)
- [Meta-Learning Dynamics Forecasting Using Task Inference](https://mlsanity.com/paper/2102.10271/) (2021)
- [Deep Learning for Reduced Order Modelling and Efficient Temporal Evolution of Fluid Simulations](https://mlsanity.com/paper/2107.04556/) (2021)
- [Transformer Meets Boundary Value Inverse Problems](https://mlsanity.com/paper/2209.14977/) (2022)
- [Deep Reinforcement Learning for Adaptive Mesh Refinement](https://mlsanity.com/paper/2209.12351/) (2022)
- [Learning Controllable Adaptive Simulation for Multi-resolution Physics](https://mlsanity.com/paper/2305.01122/) (2023)

## Built on by (most cited first, 21 of 21)

- [Unisolver: PDE-Conditional Transformers Towards Universal Neural PDE Solvers](https://mlsanity.com/paper/2405.17527/) (2024)
- [VICON: Vision In-Context Operator Networks for Multi-Physics Fluid Dynamics Prediction](https://mlsanity.com/paper/2411.16063/) (2024)
- [Strategies for Pretraining Neural Operators](https://mlsanity.com/paper/2406.08473/) (2024)
- [A Multimodal PDE Foundation Model for Prediction and Scientific Text Descriptions](https://mlsanity.com/paper/2502.06026/) (2025)
- [Predicting Change, Not States: An Alternate Framework for Neural PDE Surrogates](https://mlsanity.com/paper/2412.13074/) (2024)
- [Explain Like I'm Five: Using LLMs to Improve PDE Surrogate Models with Text](https://mlsanity.com/paper/2410.01137/) (2024)
- [PICL: Physics Informed Contrastive Learning for Partial Differential Equations](https://mlsanity.com/paper/2401.16327/) (2024)
- [Latent Neural PDE Solver: a reduced-order modelling framework for partial differential equations](https://mlsanity.com/paper/2402.17853/) (2024)
- [Hyena Neural Operator for Partial Differential Equations](https://mlsanity.com/paper/2306.16524/) (2023)
- [Flow marching for a generative PDE foundation model](https://mlsanity.com/paper/2509.18611/) (2025)
- [Latent Generative Solvers for Generalizable Long-Term Physics Simulation](https://mlsanity.com/paper/2602.11229/) (2026)
- [Generalizing PDE Emulation with Equation-Aware Neural Operators](https://mlsanity.com/paper/2511.09729/) (2025)
- [A Physics-Informed Fourier-Wavelet Transformer for Multiscale Computational Fluid Dynamics Surrogate Modeling](https://mlsanity.com/paper/2606.24696/) (2026)
- [Inpainting Computational Fluid Dynamics with Deep Learning](https://mlsanity.com/paper/2402.17185/) (2024)
- [SysCaps: Language Interfaces for Simulation Surrogates of Complex Systems](https://mlsanity.com/paper/2405.19653/) (2024)
- [Attention-Based Learning for Fluid State Interpolation and Editing in a Time-Continuous Framework](https://mlsanity.com/paper/2406.08188/) (2024)
- [FieldFormer: Locality-Aware Transformers for Spatio-Temporal Modeling on Sparse Sensor Networks](https://mlsanity.com/paper/2510.03589/) (2025)
- [In Transformer We Trust? A Perspective on Transformer Architecture Failure Modes](https://mlsanity.com/paper/2602.14318/) (2026)
- [LLT: Local Linear Transformer for PDE Operator Learning](https://mlsanity.com/paper/2607.07718/) (2026)
- [Large language models for partial differential equation workflows](https://mlsanity.com/paper/2608.03600/) (2026)
- [RD-JEPA: Predictive latent pretraining for few-trajectory transfer across reaction--diffusion equations](https://mlsanity.com/paper/2609.29403/) (2026)

## Abstract (arXiv)

Solving Partial Differential Equations (PDEs) is the core of many fields of science and engineering. While classical approaches are often prohibitively slow, machine learning models often fail to incorporate complete system information. Over the past few years, transformers have had a significant impact on the field of Artificial Intelligence and have seen increased usage in PDE applications. However, despite their success, transformers currently lack integration with physics and reasoning. This study aims to address this issue by introducing PITT: Physics Informed Token Transformer. The purpose of PITT is to incorporate the knowledge of physics by embedding partial differential equations (PDEs) into the learning process. PITT uses an equation tokenization method to learn an analytically-driven numerical update operator. By tokenizing PDEs and embedding partial derivatives, the transformer models become aware of the underlying knowledge behind physical processes. To demonstrate this, PITT is tested on challenging 1D and 2D PDE neural operator prediction tasks. The results show that PITT outperforms popular neural operator models and has the ability to extract physically relevant information from governing equations.