# Inducing Point Operator Transformer: A Flexible and Scalable Architecture for Solving PDEs

arXiv: https://arxiv.org/abs/2312.10975 · HTML: https://mlsanity.com/paper/2312.10975/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 57 references that are in the map.
- First posted on arXiv 18 Dec 2023.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [Learning to Simulate Complex Physics with Graph Networks](https://mlsanity.com/paper/2002.09405/) (2020)
- [Learning Mesh-Based Simulation with Graph Networks](https://mlsanity.com/paper/2010.03409/) (2020)
- [FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operators](https://mlsanity.com/paper/2202.11214/) (2022)
- [Neural Operator: Graph Kernel Network for Partial Differential Equations](https://mlsanity.com/paper/2003.03485/) (2020)
- [Perceiver IO: A General Architecture for Structured Inputs & Outputs](https://mlsanity.com/paper/2107.14795/) (2021)
- [Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attention](https://mlsanity.com/paper/2102.03902/) (2021)
- [Fourier Neural Operator with Learned Deformations for PDEs on General Geometries](https://mlsanity.com/paper/2207.05209/) (2022)
- [Neural Operator: Learning Maps Between Function Spaces](https://mlsanity.com/paper/2108.08481/) (2021)
- [Multipole Graph Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2006.09535/) (2020)
- [Choose a Transformer: Fourier or Galerkin](https://mlsanity.com/paper/2105.14995/) (2021)
- [GNOT: A General Neural Operator Transformer for Operator Learning](https://mlsanity.com/paper/2302.14376/) (2023)
- [Transformer for Partial Differential Equations' Operator Learning](https://mlsanity.com/paper/2205.13671/) (2022)
- [Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformers](https://mlsanity.com/paper/2111.13587/) (2021)
- [Multiwavelet-based Operator Learning for Differential Equations](https://mlsanity.com/paper/2109.13459/) (2021)
- [Dataset Meta-Learning from Kernel Ridge-Regression](https://mlsanity.com/paper/2011.00050/) (2020)
- [Transformer Dissection: A Unified Understanding of Transformer's Attention via the Lens of Kernel](https://mlsanity.com/paper/1908.11775/) (2019)
- [MeshfreeFlowNet: A Physics-Constrained Deep Continuous Space-Time Super-Resolution Framework](https://mlsanity.com/paper/2005.01463/) (2020)
- [Continuous PDE Dynamics Forecasting with Implicit Neural Representations](https://mlsanity.com/paper/2209.14855/) (2022)
- [The Sensory Neuron as a Transformer: Permutation-Invariant Neural Networks for Reinforcement Learning](https://mlsanity.com/paper/2109.02869/) (2021)
- [A Point-Cloud Deep Learning Framework for Prediction of Fluid Flow Fields on Irregular Geometries](https://mlsanity.com/paper/2010.09469/) (2020)

## Built on by (most cited first, 18 of 18)

- [Physics-Informed Geometry-Aware Neural Operator](https://mlsanity.com/paper/2408.01600/) (2024)
- [Deep Learning and Foundation Models for Weather Prediction: A Survey](https://mlsanity.com/paper/2501.06907/) (2025)
- [Structure-Preserving Learning Improves Geometry Generalization in Neural PDEs](https://mlsanity.com/paper/2602.02788/) (2026)
- [SAOT: An Enhanced Locality-Aware Spectral Transformer for Solving PDEs](https://mlsanity.com/paper/2511.18777/) (2025)
- [SCENT: Robust Spatiotemporal Learning for Continuous Scientific Data via Scalable Conditioned Neural Fields](https://mlsanity.com/paper/2504.12262/) (2025)
- [PDEfuncta: Spectrally-Aware Neural Representation for PDE Solution Modeling](https://mlsanity.com/paper/2506.12790/) (2025)
- [Learning Lagrangian Interaction Dynamics with Sampling-Based Model Order Reduction](https://mlsanity.com/paper/2407.03925/) (2024)
- [GeoAggregator: An Efficient Transformer Model for Geo-Spatial Tabular Data](https://mlsanity.com/paper/2502.15032/) (2025)
- [MMET: A Multi-Input and Multi-Scale Transformer for Efficient PDEs Solving](https://mlsanity.com/paper/2506.17230/) (2025)
- [Learning Neural Operators from Partial Observations via Latent Autoregressive Modeling](https://mlsanity.com/paper/2601.15547/) (2026)
- [From Points to Edges: Edge-Conditioned Spectral Operators for Physics-Sensitive PDE Learning](https://mlsanity.com/paper/2608.06894/) (2026)
- [From Uniform to Adaptive: General Skip-Block Mechanisms for Efficient PDE Neural Operators](https://mlsanity.com/paper/2511.00032/) (2025)
- [PGOT: A Physics-Geometry Operator Transformer for Complex PDEs](https://mlsanity.com/paper/2512.23192/) (2025)
- [PGD-NO: A Neural Operator with Precomputed Geometry Decomposition for 3D Million-scale Physics Simulations](https://mlsanity.com/paper/2607.08025/) (2026)
- [HiLNO: A Hierarchical Latent Neural Operator with Multi-Scale Supervision for PDEs on General Geometries](https://mlsanity.com/paper/2609.18419/) (2026)
- [Function-Space Transformer with Adaptive Anchors](https://mlsanity.com/paper/2609.38348/) (2026)
- [Refined Grid Sampling for Efficient Implicit Neural Representations of Large-Scale Scientific Data](https://mlsanity.com/paper/2610.14205/) (2026)
- [Learnable Hodge-Laplacian Driven Mixture of Experts for Steady-PDE Surrogates on Coarse Unstructured Meshes](https://mlsanity.com/paper/2610.20585/) (2026)

## Abstract (arXiv)

Solving partial differential equations (PDEs) by learning the solution operators has emerged as an attractive alternative to traditional numerical methods. However, implementing such architectures presents two main challenges: flexibility in handling irregular and arbitrary input and output formats and scalability to large discretizations. Most existing architectures are limited by their desired structure or infeasible to scale large inputs and outputs. To address these issues, we introduce an attention-based model called an inducing-point operator transformer (IPOT). Inspired by inducing points methods, IPOT is designed to handle any input function and output query while capturing global interactions in a computationally efficient way. By detaching the inputs/outputs discretizations from the processor with a smaller latent bottleneck, IPOT offers flexibility in processing arbitrary discretizations and scales linearly with the size of inputs/outputs. Our experimental results demonstrate that IPOT achieves strong performances with manageable computational complexity on an extensive range of PDE benchmarks and real-world weather forecasting scenarios, compared to state-of-the-art methods.