# Positional Knowledge is All You Need: Position-induced Transformer (PiT) for Operator Learning

arXiv: https://arxiv.org/abs/2405.09285 · HTML: https://mlsanity.com/paper/2405.09285/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 56 references that are in the map.
- First posted on arXiv 15 May 2024.

## Builds on (most cited first, 30 of 46)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Neural Operator: Graph Kernel Network for Partial Differential Equations](https://mlsanity.com/paper/2003.03485/) (2020)
- [Open3D: A Modern Library for 3D Data Processing](https://mlsanity.com/paper/1801.09847/) (2018)
- [DGM: A deep learning algorithm for solving partial differential equations](https://mlsanity.com/paper/1708.07469/) (2017)
- [A Generalization of Transformer Networks to Graphs](https://mlsanity.com/paper/2012.09699/) (2020)
- [The Deep Ritz method: A deep learning-based numerical algorithm for solving variational problems](https://mlsanity.com/paper/1710.00211/) (2017)
- [Graph Transformer Networks](https://mlsanity.com/paper/1911.06455/) (2019)
- [Rethinking Graph Transformers with Spectral Attention](https://mlsanity.com/paper/2106.03893/) (2021)
- [Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attention](https://mlsanity.com/paper/2102.03902/) (2021)
- [Fourier Neural Operator with Learned Deformations for PDEs on General Geometries](https://mlsanity.com/paper/2207.05209/) (2022)
- [Random Feature Attention](https://mlsanity.com/paper/2103.02143/) (2021)
- [Multipole Graph Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2006.09535/) (2020)
- [Choose a Transformer: Fourier or Galerkin](https://mlsanity.com/paper/2105.14995/) (2021)
- [GNOT: A General Neural Operator Transformer for Operator Learning](https://mlsanity.com/paper/2302.14376/) (2023)
- [PDE-Net: Learning PDEs from Data](https://mlsanity.com/paper/1710.09668/) (2017)
- [Transformer for Partial Differential Equations' Operator Learning](https://mlsanity.com/paper/2205.13671/) (2022)
- [Model Reduction and Neural Networks for Parametric PDEs](https://mlsanity.com/paper/2005.03180/) (2020)
- [Multiwavelet-based Operator Learning for Differential Equations](https://mlsanity.com/paper/2109.13459/) (2021)
- [Neural Operators for Accelerating Scientific Simulations and Design](https://mlsanity.com/paper/2309.15325/) (2023)
- [PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid Deep Network](https://mlsanity.com/paper/1812.04426/) (2018)
- [Structure-Aware Transformer for Graph Representation Learning](https://mlsanity.com/paper/2202.03036/) (2022)

## Built on by (most cited first, 17 of 17)

- [Transolver is a Linear Transformer: Revisiting Physics-Attention through the Lens of Linear Attention](https://mlsanity.com/paper/2511.06294/) (2025)
- [DUE: A Deep Learning Framework and Library for Modeling Unknown Equations](https://mlsanity.com/paper/2504.10373/) (2025)
- [SAOT: An Enhanced Locality-Aware Spectral Transformer for Solving PDEs](https://mlsanity.com/paper/2511.18777/) (2025)
- [SMART: Scalable Mesh-free Aerodynamic Simulations from Raw Geometries using a Transformer-based Surrogate Model](https://mlsanity.com/paper/2601.18707/) (2026)
- [Implicit factorized transformer approach to fast prediction of turbulent channel flows](https://mlsanity.com/paper/2412.18840/) (2024)
- [GITO: Graph-Informed Transformer Operator for Learning Complex Partial Differential Equations](https://mlsanity.com/paper/2506.13906/) (2025)
- [Function graph transformers universally approximate operators between function spaces](https://mlsanity.com/paper/2605.17968/) (2026)
- [LVM-GP: Uncertainty-Aware PDE Solver via coupling latent variable model and Gaussian process](https://mlsanity.com/paper/2507.22493/) (2025)
- [Can Data-Driven Dynamics Reveal Hidden Physics? There Is A Need for Interpretable Neural Operators](https://mlsanity.com/paper/2510.02683/) (2025)
- [From Points to Edges: Edge-Conditioned Spectral Operators for Physics-Sensitive PDE Learning](https://mlsanity.com/paper/2608.06894/) (2026)
- [Graph Neural Networks as a Substitute for Transformers in Single-Cell Transcriptomics](https://mlsanity.com/paper/2507.04125/) (2025)
- [GeoFunFlow: Geometric Function Flow Matching for Inverse Operator Learning over Complex Geometries](https://mlsanity.com/paper/2509.24117/) (2025)
- [MAD-NG: Meta-Auto-Decoder Neural Galerkin Method for Solving Parametric Partial Differential Equations](https://mlsanity.com/paper/2512.21633/) (2025)
- [Flux-form spatiotemporal neural operators for coarse-grained dynamics of multiscale PDEs](https://mlsanity.com/paper/2608.18148/) (2026)
- [Disentangling Attention in Deep Operator Learning: A Controlled Study of Data-Driven and Physics-Informed Architectures](https://mlsanity.com/paper/2609.04407/) (2026)
- [HiLNO: A Hierarchical Latent Neural Operator with Multi-Scale Supervision for PDEs on General Geometries](https://mlsanity.com/paper/2609.18419/) (2026)
- [Neural Scaling Laws of Transformer Operator Network](https://mlsanity.com/paper/2609.33533/) (2026)

## Abstract (arXiv)

Operator learning for Partial Differential Equations (PDEs) is rapidly emerging as a promising approach for surrogate modeling of intricate systems. Transformers with the self-attention mechanismunicodex2013a powerful tool originally designed for natural language processingunicodex2013have recently been adapted for operator learning. However, they confront challenges, including high computational demands and limited interpretability. This raises a critical question: Is there a more efficient attention mechanism for Transformer-based operator learning? This paper proposes the Position-induced Transformer (PiT), built on an innovative position-attention mechanism, which demonstrates significant advantages over the classical self-attention in operator learning. Position-attention draws inspiration from numerical methods for PDEs. Different from self-attention, position-attention is induced by only the spatial interrelations of sampling positions for input functions of the operators, and does not rely on the input function values themselves, thereby greatly boosting efficiency. PiT exhibits superior performance over current state-of-the-art neural operators in a variety of complex operator learning tasks across diverse PDE benchmarks. Additionally, PiT possesses an enhanced discretization convergence feature, compared to the widely-used Fourier neural operator.