mlsanityData to 14 Oct 2026

Transformer for Partial Differential Equations' Operator Learning

arXiv 2205.13671 ↗First posted 26 May 2022cs.LG cs.AIcited by 283

Not on the map. Why?

Cited by 24 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Data-driven learning of partial differential equations' solution operators has recently emerged as a promising paradigm for approximating the underlying solutions. The solution operators are usually parameterized by deep learning models that are built upon problem-specific inductive biases. An example is a convolutional or a graph neural network that exploits the local grid structure where functions' values are sampled. The attention mechanism, on the other hand, provides a flexible way to implicitly exploit the patterns within inputs, and furthermore, relationship between arbitrary query locations and inputs. In this work, we present an attention-based framework for data-driven operator learning, which we term Operator Transformer (OFormer). Our framework is built upon self-attention, cross-attention, and a set of point-wise multilayer perceptrons (MLPs), and thus it makes few assumptions on the sampling pattern of the input function or query locations. We show that the proposed framework is competitive on standard benchmark problems and can flexibly be adapted to randomly sampled input.

Key facts. Cited by 283 papers in the map (data through 14 Oct 2026). Builds on 68 of its 100 references that are in the map. First posted on arXiv 26 May 2022.

Builds on

68 of 100 references.

References · 68

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  7. 2017Graph Attention Networkscited by 8,923
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  9. 2016Layer Normalizationcited by 5,669
  10. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  11. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  12. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
38 more
  1. 2017Neural Message Passing for Quantum Chemistrycited by 3,487
  2. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  3. 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,908
  4. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,200
  5. 2019DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operatorscited by 2,018
  6. 2020Rethinking Attention with Performerscited by 1,601
  7. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,198
  8. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  9. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,026
  10. 2020Learning to Simulate Complex Physics with Graph Networkscited by 880
  11. 2020Learning Mesh-Based Simulation with Graph Networkscited by 868
  12. 2022FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operatorscited by 776
  13. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  14. 2020Neural Operator: Graph Kernel Network for Partial Differential Equationscited by 734
  15. 2016Language Modeling with Gated Convolutional Networkscited by 702
  16. 2017DGM: A deep learning algorithm for solving partial differential equationscited by 678
  17. 2014Neural Turing Machinescited by 644
  18. 2021Characterizing possible failure modes in physics-informed neural networkscited by 643
  19. 2019DeepXDE: A deep learning library for solving differential equationscited by 619
  20. 2017The Deep Ritz method: A deep learning-based numerical algorithm for solving variational problemscited by 583
  21. 2021Learning the solution operator of parametric partial differential equations with physics-informed DeepOnetscited by 540
  22. 2021Physics-Informed Neural Operator for Learning Partial Differential Equationscited by 527
  23. 2017Deep learning for universal linear embeddings of nonlinear dynamicscited by 494
  24. 2021Neural Operator: Learning Maps Between Function Spacescited by 414
  25. 2022Message Passing Neural PDE Solverscited by 386
  26. 2019Machine Learning for Fluid Mechanicscited by 385
  27. 2020Multipole Graph Neural Operator for Parametric Partial Differential Equationscited by 349
  28. 2021Choose a Transformer: Fourier or Galerkincited by 343
  29. 2021U-FNO -- An enhanced Fourier neural operator-based deep-learning model for multiphase flowcited by 297
  30. 2020On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networkscited by 284
  31. 2020Model Reduction and Neural Networks for Parametric PDEscited by 258
  32. 2021Factorized Fourier Neural Operatorscited by 249
  33. 2021Multiwavelet-based Operator Learning for Differential Equationscited by 234
  34. 2022U-NO: U-shaped Neural Operatorscited by 230
  35. 2019Towards Physics-informed Deep Learning for Turbulent Flow Predictioncited by 204
  36. 2017Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Ratescited by 199
  37. 2018Bayesian Deep Convolutional Encoder-Decoder Networks for Surrogate Modeling and Uncertainty Quantificationcited by 199
  38. 2017Learning Koopman Invariant Subspaces for Dynamic Mode Decompositioncited by 158

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 9; 2023: 29; 2024: 71; 2025: 90; 2026: 84.

283 papers.

Sort:

By area:

  1. 2023GNOT: A General Neural Operator Transformer for Operator Learningcited by 336
  2. 2024Transolver: A Fast Transformer Solver for PDEs on General Geometriescited by 326
  3. 2023Geometry-Informed Neural Operator for Large-Scale 3D PDEscited by 303
  4. 2023Neural Operators for Accelerating Scientific Simulations and Designcited by 221
  5. 2024Poseidon: Efficient Foundation Models for PDEscited by 200
  6. 2022Towards Multi-spatiotemporal-scale Generalized PDE Modelingcited by 197
  7. 2022A Physics-informed Diffusion Model for High-fidelity Flow Field Reconstructioncited by 152
  8. 2024DPOT: Auto-Regressive Denoising Operator Transformer for Large-Scale PDE Pre-Trainingcited by 151
  9. 2023Scalable Transformer for PDE Surrogate Modelingcited by 128
  10. 2024DiffusionPDE: Generative PDE-Solving Under Partial Observationcited by 108
  11. 2022Clifford Neural Layers for PDE Modelingcited by 92
  12. 2022Physics-Informed Machine Learning: A Survey on Problems, Methods and Applicationscited by 92
38 more
  1. 2023Operator Learning with Neural Fields: Tackling PDEs on General Geometriescited by 84
  2. 2024Neural Operators with Localized Integral and Differential Kernelscited by 62
  3. 2023Improved Operator Learning by Orthogonal Attentioncited by 61
  4. 2025AB-UPT: Scaling Neural CFD Surrogates for High-Fidelity Automotive Aerodynamics Simulations via Anchored-Branched Universal Physics Transformerscited by 53
  5. 2024CViT: Continuous Vision Transformer for Operator Learningcited by 51
  6. 2023Training neural operators to preserve invariant measures of chaotic attractorscited by 46
  7. 2022Efficient Learning of Mesh-Based Physical Simulation with BSMS-GNNcited by 45
  8. 2022Mitigating spectral bias for the multiscale operator learningcited by 44
  9. 2024Continuum Attention for Neural Operatorscited by 40
  10. 2024AROMA: Preserving Spatial Structure for Latent PDE Modeling with Local Neural Fieldscited by 39
  11. 2025CodePDE: An Inference Framework for LLM-driven PDE Solver Generationcited by 39
  12. 2023Domain Agnostic Fourier Neural Operatorscited by 38
  13. 2024Integrating Neural Operators with Diffusion Models Improves Spectral Representation in Turbulence Modelingcited by 36
  14. 2024Transformers as Neural Operators for Solutions of Differential Equations with Finite Regularitycited by 34
  15. 2023Out-of-distributional risk bounds for neural operators with applications to the Helmholtz equationcited by 32
  16. 2025PDE-Transformer: Efficient and Versatile Transformers for Physics Simulationscited by 32
  17. 2024On the Benefits of Memory for Modeling Time-Dependent PDEscited by 31
  18. 2025Mitigating Spectral Bias in Neural Operators via High-Frequency Scaling for Physical Systemscited by 31
  19. 2023ViTO: Vision Transformer-Operatorcited by 29
  20. 2024Wavelet Diffusion Neural Operatorcited by 29
  21. 2024State-space models are accurate and efficient neural operators for dynamical systemscited by 28
  22. 2025Generative Latent Neural PDE Solver using Flow Matchingcited by 28
  23. 2025Physics vs Distributions: Pareto Optimal Flow Matching with Physics Constraintscited by 28
  24. 2024Universal Physics Transformers: A Framework For Efficiently Scaling Neural Operatorscited by 27
  25. 2024Differentiability in Unrolled Training of Neural Physics Simulators on Transient Dynamicscited by 27
  26. 2024UPS: Efficiently Building Foundation Models for PDE Solving via Cross-Modal Adaptationcited by 27
  27. 2024VICON: Vision In-Context Operator Networks for Multi-Physics Fluid Dynamics Predictioncited by 25
  28. 2023Deep Operator Network Approximation Rates for Lipschitz Operatorscited by 24
  29. 2024HAMLET: Graph Transformer Neural Operator for Partial Differential Equationscited by 24
  30. 2022Transformer Meets Boundary Value Inverse Problemscited by 22
  31. 2025Erwin: A Tree-based Hierarchical Transformer for Large-scale Physical Systemscited by 22
  32. 2025PhysiX: A Foundation Model for Physics Simulationscited by 22
  33. 2022Neural and spectral operator surrogates: unified construction and expression rate boundscited by 21
  34. 2023Machine Learning for Partial Differential Equationscited by 21
  35. 2023Physics Informed Token Transformer for Solving Partial Differential Equationscited by 21
  36. 2025Neural Chaos: A Spectral Stochastic Neural Operatorcited by 21
  37. 2026Transolver-3: Scaling Up Transformer Solvers to Industrial-Scale Geometriescited by 20
  38. 2023Deep Equilibrium Based Neural Operators for Steady-State PDEscited by 19

50 most cited of 283 All 283 (JSON)