mlsanityData to 14 Oct 2026

Scalable Transformer for PDE Surrogate Modeling

arXiv 2305.17560 ↗First posted 27 May 2023cs.LGcited by 128

Not on the map. Why?

Abstract (arXiv)

Transformer has shown state-of-the-art performance on various applications and has recently emerged as a promising tool for surrogate modeling of partial differential equations (PDEs). Despite the introduction of linear-complexity attention, applying Transformer to problems with a large number of grid points can be numerically unstable and computationally expensive. In this work, we propose Factorized Transformer (FactFormer), which is based on an axial factorized kernel integral. Concretely, we introduce a learnable projection operator that decomposes the input function into multiple sub-functions with one-dimensional domain. These sub-functions are then evaluated and used to compute the instance-based kernel with an axial factorized scheme. We showcase that the proposed model is able to simulate 2D Kolmogorov flow on a 256× 256 grid and 3D smoke buoyancy on a 64×64×64 grid with good accuracy and efficiency. The proposed factorized scheme can serve as a computationally efficient low-rank surrogate for the full attention scheme when dealing with multi-dimensional problems.

Key facts. Cited by 128 papers in the map (data through 14 Oct 2026). Builds on 85 of its 123 references that are in the map. First posted on arXiv 27 May 2023.

Builds on

85 of 123 references.

References · 85

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  7. 2020End-to-End Object Detection with Transformerscited by 7,414
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  9. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  10. 2020Longformer: The Long-Document Transformercited by 3,413
  11. 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,908
  12. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,200
38 more
  1. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  2. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  3. 2020Reformer: The Efficient Transformercited by 1,668
  4. 2020Rethinking Attention with Performerscited by 1,601
  5. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  6. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  7. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,198
  8. 2015Multi-Scale Context Aggregation by Dilated Convolutionscited by 1,133
  9. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,026
  10. 2020Learning to Simulate Complex Physics with Graph Networkscited by 880
  11. 2020Learning Mesh-Based Simulation with Graph Networkscited by 868
  12. 2022FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operatorscited by 776
  13. 2020Neural Operator: Graph Kernel Network for Partial Differential Equationscited by 734
  14. 2014Neural Turing Machinescited by 644
  15. 2019DeepXDE: A deep learning library for solving differential equationscited by 619
  16. 2017Solving high-dimensional partial differential equations using deep learningcited by 556
  17. 2021Learning the solution operator of parametric partial differential equations with physics-informed DeepOnetscited by 540
  18. 2021Physics-Informed Neural Operator for Learning Partial Differential Equationscited by 527
  19. 2021Physics-informed neural networks (PINNs) for fluid mechanics: A reviewcited by 473
  20. 2021Machine learning accelerated computational fluid dynamicscited by 471
  21. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 471
  22. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 419
  23. 2021Neural Operator: Learning Maps Between Function Spacescited by 414
  24. 2022Message Passing Neural PDE Solverscited by 386
  25. 2018Efficient Attention: Attention with Linear Complexitiescited by 368
  26. 2019Axial Attention in Multidimensional Transformerscited by 358
  27. 2020Multipole Graph Neural Operator for Parametric Partial Differential Equationscited by 349
  28. 2021Choose a Transformer: Fourier or Galerkincited by 343
  29. 2023ClimaX: A foundation model for weather and climatecited by 337
  30. 2023GNOT: A General Neural Operator Transformer for Operator Learningcited by 336
  31. 2021U-FNO -- An enhanced Fourier neural operator-based deep-learning model for multiphase flowcited by 297
  32. 2022Transformer for Partial Differential Equations' Operator Learningcited by 283
  33. 2020WeatherBench: A benchmark dataset for data-driven weather forecastingcited by 275
  34. 2022GraphCast: Learning skillful medium-range global weather forecastingcited by 267
  35. 2020Model Reduction and Neural Networks for Parametric PDEscited by 258
  36. 2015Tensorizing Neural Networkscited by 250
  37. 2021Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformerscited by 248
  38. 2021Multiwavelet-based Operator Learning for Differential Equationscited by 234

50 most cited of 85 All 85 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 35; 2025: 47; 2026: 43.

128 papers.

Sort:

By area:

  1. 2024Transolver: A Fast Transformer Solver for PDEs on General Geometriescited by 326
  2. 2023Improved Operator Learning by Orthogonal Attentioncited by 61
  3. 2024Pretraining Codomain Attention Neural Operators for Solving Multiphysics PDEscited by 59
  4. 2025AB-UPT: Scaling Neural CFD Surrogates for High-Fidelity Automotive Aerodynamics Simulations via Anchored-Branched Universal Physics Transformerscited by 53
  5. 2024Unisolver: PDE-Conditional Transformers Towards Universal Neural PDE Solverscited by 35
  6. 2025PDE-Transformer: Efficient and Versatile Transformers for Physics Simulationscited by 32
  7. 2024On the Benefits of Memory for Modeling Time-Dependent PDEscited by 31
  8. 2024Recent Advances on Machine Learning for Computational Fluid Dynamics: A Surveycited by 28
  9. 2025Generative Latent Neural PDE Solver using Flow Matchingcited by 28
  10. 2024Text2PDE: Latent Diffusion Models for Accessible Physics Simulationcited by 25
  11. 2025Erwin: A Tree-based Hierarchical Transformer for Large-scale Physical Systemscited by 22
  12. 2025Neural Chaos: A Spectral Stochastic Neural Operatorcited by 21
38 more
  1. 2026Transolver-3: Scaling Up Transformer Solvers to Industrial-Scale Geometriescited by 20
  2. 2023Multi-scale Time-stepping of Partial Differential Equations with Transformerscited by 14
  3. 2024Strategies for Pretraining Neural Operatorscited by 14
  4. 2024P2C2Net: PDE-Preserved Coarse Correction Network for efficient prediction of spatiotemporal dynamicscited by 13
  5. 2024Vectorized Conditional Neural Fields: A Framework for Solving Time-dependent Parametric Partial Differential Equationscited by 12
  6. 2025Machine learning for modelling unstructured grid data in computational physics: a reviewcited by 12
  7. 2025Principled Approaches for Extending Neural Architectures to Function Spaces for Operator Learningcited by 12
  8. 2024Hierarchical Neural Operator Transformer with Learnable Frequency-aware Loss Prior for Arbitrary-scale Super-resolutioncited by 11
  9. 2024Spectral-Refiner: Accurate Fine-Tuning of Spatiotemporal Fourier Neural Operator for Turbulent Flowscited by 11
  10. 2024Holistic Physics Solver: Learning PDEs in a Unified Spectral-Physical Spacecited by 10
  11. 2025Latent Mamba Operator for Partial Differential Equationscited by 10
  12. 2026GeoPT: Scaling Physics Simulation via Lifted Geometric Pre-Trainingcited by 10
  13. 2024OmniArch: Building Foundation Model For Scientific Computingcited by 9
  14. 2024FUSE: Fast Unified Simulation and Estimation for PDEscited by 8
  15. 2024Predicting Change, Not States: An Alternate Framework for Neural PDE Surrogatescited by 8
  16. 2025TANTE: Time-Adaptive Operator Learning via Neural Taylor Expansioncited by 8
  17. 2025AMR-Transformer: Enabling Efficient Long-range Interaction for Complex Neural Fluid Simulationcited by 8
  18. 2025FEDONet : Fourier-Embedded DeepONet for Spectrally Accurate Operator Learningcited by 8
  19. 2026Structure-Preserving Learning Improves Geometry Generalization in Neural PDEscited by 8
  20. 2023HelmFluid: Learning Helmholtz Dynamics for Interpretable Fluid Predictioncited by 7
  21. 2024Explain Like I'm Five: Using LLMs to Improve PDE Surrogate Models with Textcited by 7
  22. 2024CaFA: Global Weather Forecasting with Factorized Attention on Spherecited by 6
  23. 2025FLARE: Fast Low-rank Attention Routing Enginecited by 6
  24. 2025EddyFormer: Accelerated Neural Simulations of Three-Dimensional Turbulence at Scalecited by 6
  25. 2025Neural Green's Functionscited by 6
  26. 2024PICL: Physics Informed Contrastive Learning for Partial Differential Equationscited by 5
  27. 2024Latent Neural PDE Solver: a reduced-order modelling framework for partial differential equationscited by 5
  28. 2024Spatio-Temporal Fluid Dynamics Modeling via Physical-Awareness and Parameter Diffusion Guidancecited by 5
  29. 2024Reference Neural Operators: Learning the Smooth Dependence of Solutions of PDEs on Geometric Deformationscited by 5
  30. 2024Separable Operator Networkscited by 5
  31. 2024Pretraining a Neural Operator in Lower Dimensionscited by 5
  32. 2024Latent Neural Operator Pretraining for Solving Time-Dependent PDEscited by 5
  33. 2024PACE: Pacing Operator Learning to Accurate Optical Field Simulation for Complicated Photonic Devicescited by 5
  34. 2025PeSANet: Physics-encoded Spectral Attention Network for Simulating PDE-Governed Complex Systemscited by 5
  35. 2025Physics-informed Temporal Alignment for Auto-regressive PDE Foundation Modelscited by 5
  36. 2025FD-Bench: A Modular and Fair Benchmark for Data-driven Fluid Simulationcited by 5
  37. 2025Overtone: Cyclic Patch Modulation for Clean, Efficient, and Flexible Physics Emulatorscited by 5
  38. 2026DiffusionRollout: Uncertainty-Aware Rollout Planning in Long-Horizon PDE Solvingcited by 5

50 most cited of 128 All 128 (JSON)