mlsanityData to 14 Oct 2026

PDE-Transformer: Efficient and Versatile Transformers for Physics Simulations

arXiv 2505.24717 ↗First posted 30 May 2025cs.LGcited by 32

Not on the map. Why?

Abstract (arXiv)

We introduce PDE-Transformer, an improved transformer-based architecture for surrogate modeling of physics simulations on regular grids. We combine recent architectural improvements of diffusion transformers with adjustments specific for large-scale simulations to yield a more scalable and versatile general-purpose transformer architecture, which can be used as the backbone for building large-scale foundation models in physical sciences. We demonstrate that our proposed architecture outperforms state-of-the-art transformer architectures for computer vision on a large dataset of 16 different types of PDEs. We propose to embed different physical channels individually as spatio-temporal tokens, which interact via channel-wise self-attention. This helps to maintain a consistent information density of tokens when learning multiple types of PDEs simultaneously. We demonstrate that our pre-trained models achieve improved performance on several challenging downstream tasks compared to training from scratch and also beat other foundation model architectures for physics simulations.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Builds on 65 of its 92 references that are in the map. First posted on arXiv 30 May 2025.

Builds on

65 of 92 references.

References · 65

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  7. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  10. 2022Scalable Diffusion Models with Transformerscited by 6,054
  11. 2022Flow Matching for Generative Modelingcited by 5,702
  12. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
38 more
  1. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  2. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,333
  3. 2019DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operatorscited by 2,018
  4. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  5. 2019Root Mean Square Layer Normalizationcited by 1,480
  6. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,228
  7. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  8. 2020Neural Operator: Graph Kernel Network for Partial Differential Equationscited by 734
  9. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 694
  10. 2021Physics-Informed Neural Operator for Learning Partial Differential Equationscited by 527
  11. 2022All are Worth Words: A ViT Backbone for Diffusion Modelscited by 502
  12. 2021Machine learning accelerated computational fluid dynamicscited by 471
  13. 2023Simple diffusion: End-to-end diffusion for high resolution imagescited by 422
  14. 2019Axial Attention in Multidimensional Transformerscited by 358
  15. 2021Choose a Transformer: Fourier or Galerkincited by 343
  16. 2024Transolver: A Fast Transformer Solver for PDEs on General Geometriescited by 326
  17. 2022Transformer for Partial Differential Equations' Operator Learningcited by 283
  18. 2024Foundation Models for Time Series Analysis: A Tutorial and Surveycited by 239
  19. 2023LCM-LoRA: A Universal Stable-Diffusion Acceleration Modulecited by 211
  20. 2020Solver-in-the-Loop: Learning from Differentiable Physics to Interact with Iterative PDE-Solverscited by 205
  21. 2022Towards Multi-spatiotemporal-scale Generalized PDE Modelingcited by 197
  22. 2024The Well: a Large-Scale Collection of Diverse Physics Simulations for Machine Learningcited by 145
  23. 2023Continual Diffusion: Continual Customization of Text-to-Image Diffusion with C-LoRAcited by 141
  24. 2023Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer Behaviorcited by 131
  25. 2023Scalable Transformer for PDE Surrogate Modelingcited by 128
  26. 2018Deep Fluids: A Generative Network for Parameterized Fluid Simulationscited by 121
  27. 2023The Expressive Power of Low-Rank Adaptationcited by 102
  28. 2023In-Context Operator Learning with Data Prompts for Differential Equation Problemscited by 101
  29. 2023Multiple Physics Pretraining for Physical Surrogate Modelscited by 98
  30. 2022Deep transfer operator learning for partial differential equations under conditional shiftcited by 83
  31. 2023Benchmarking Autoregressive Conditional Diffusion Models for Turbulent Flow Simulationcited by 71
  32. 2024Tracking Meets LoRA: Faster Training, Larger Model, Stronger Performancecited by 70
  33. 2021Physics-based Deep Learningcited by 68
  34. 2024A Survey on LoRA of Large Language Modelscited by 67
  35. 2024APEBench: A Benchmark for Autoregressive Neural Emulators of PDEscited by 60
  36. 2022Neural Galerkin Schemes with Active Learning for High-Dimensional Evolution Equationscited by 56
  37. 2024U-DiTs: Downsample Tokens in U-Shaped Diffusion Transformerscited by 54
  38. 2022Learning to correct spectral methods for simulating turbulent flowscited by 49

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 25.

32 papers.

Sort:

By area:

  1. 2025Physics vs Distributions: Pareto Optimal Flow Matching with Physics Constraintscited by 28
  2. 2025MORPH: PDE Foundation Models with Arbitrary Data Modalitycited by 14
  3. 2025Neural Emulator Superiority: When Machine Learning for PDEs Surpasses its Training Datacited by 6
  4. 2026(Sparse) Attention to the Details: Preserving Spectral Fidelity in ML-based Weather Forecasting Modelscited by 5
  5. 2026Emergent Transfer of a Physics Foundation Model from Simulation to Laboratory Turbulencecited by 4
  6. 2026On the Generalization in Topology Optimization via Sensitivity-Conditioned Bernoulli Flow Matchingcited by 4
  7. 2025SuperWing: a comprehensive transonic wing dataset for data-driven aerodynamic designcited by 3
  8. 2026Out-of-distribution generalization of deep-learning surrogates for 2D PDE-generated dynamics in the small-data regimecited by 3
  9. 2026Latent Generative Solvers for Generalizable Long-Term Physics Simulationcited by 3
  10. 2026ClothTransformer: Unified Latent-Space Transformers for Scalable Cloth Simulationcited by 3
  11. 2025Efficient Generative Transformer Operators For Million-Point PDEscited by 2
  12. 2025A Mechanistic Analysis of Transformers for Dynamical Systemscited by 2
20 more
  1. 2026Iterative Refinement Neural Operators are Learned Fixed-Point Solvers: A Principled Approach to Spectral Bias Mitigationcited by 2
  2. 2026Learning Where to Simulate: Generative Active Sampling for Online PDE Surrogate Trainingcited by 2
  3. 2026TRIE: An Evaluation Framework for Stochastic PDE Surrogatescited by 2
  4. 2026StocBench: A Benchmark for Generative Modeling of Stochastic Dynamicscited by 2
  5. 2025Genomic Next-Token Predictors are In-Context Learnerscited by 1
  6. 2026Neural-HSS: Hierarchical Semi-Separable Neural PDE Solvercited by 1
  7. 2026Factorized Neural Implicit DMD for Parametric Dynamicscited by 1
  8. 2026Extending Fourier Neural Operators for Modeling Parameterized and Coupled PDEscited by 1
  9. 2026Unsupervised Adaptation of PDE Foundation Modelscited by 1
  10. 2026Sim+Real: Joint Simulation-Experiment Training Improves Balanced Prediction in Physical Systemscited by 1
  11. 2026Hierarchical Inference and Closure Learning via Adaptive Surrogates for ODEs and PDEscited by 0
  12. 2026Towards a Foundation-Model Paradigm for Aerodynamic Prediction in Three-dimensional Designcited by 0
  13. 2026Inpainting physics: self-supervised learning for context-driven fluid simulationcited by 0
  14. 2026DiffUNet^2: Bidirectional Conditional Diffusion for Probabilistic Scientific Spatiotemporal Modelingcited by 0
  15. 2026RD-JEPA: Predictive latent pretraining for few-trajectory transfer across reaction--diffusion equationscited by 0
  16. 2026Scale-Split Neural Operator for Memory- and Data-Efficient 3D Turbulence Predictioncited by 0
  17. 2026Geometry Meets Physics: Data-Efficient Pre-Training for Unstructured Neural PDE Solverscited by 0
  18. 2026EC-EarthFlow: Probabilistic emulation of daily transient global climate model simulations with flow matchingcited by 0
  19. 2026One-Way Coupling via Patchwise Adaptive Normalization in Transformer Surrogatescited by 0
  20. 2026Physics Foundation Models for Efficient Control of Fluid Flowscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025P3D: Scalable Neural Surrogates for High-Resolution 3D Physics Simulations with Global Context
  2. 2026When Attention Beats Fourier: Multi-Scale Transformers for PDE Solving on Irregular Domains
  3. 2025Video Prediction of Dynamic Physical Simulations With Pixel-Space Spatiotemporal Transformers
  4. 2025Flow marching for a generative PDE foundation model
  5. 2025Universal Physics Simulation: A Foundational Diffusion Approach
  6. 2024Text2PDE: Latent Diffusion Models for Accessible Physics Simulation
  7. 2026PDE-SSM: A Spectral State Space Approach to Spatial Mixing in Diffusion Transformers
  8. 2025Exploring Diffusion Transformer Designs via Grafting