# Universal Physics Transformers: A Framework For Efficiently Scaling Neural Operators

arXiv: https://arxiv.org/abs/2402.12365 · HTML: https://mlsanity.com/paper/2402.12365/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 64 of its 113 references that are in the map.
- First posted on arXiv 19 Feb 2024.

## Builds on (most cited first, 30 of 64)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Relational inductive biases, deep learning, and graph networks](https://mlsanity.com/paper/1806.01261/) (2018)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [Learning to Simulate Complex Physics with Graph Networks](https://mlsanity.com/paper/2002.09405/) (2020)
- [Learning Mesh-Based Simulation with Graph Networks](https://mlsanity.com/paper/2010.03409/) (2020)
- [Neural Operator: Graph Kernel Network for Partial Differential Equations](https://mlsanity.com/paper/2003.03485/) (2020)
- [ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders](https://mlsanity.com/paper/2301.00808/) (2023)
- [E(3)-Equivariant Graph Neural Networks for Data-Efficient and Accurate Interatomic Potentials](https://mlsanity.com/paper/2101.03164/) (2021)
- [Perceiver IO: A General Architecture for Structured Inputs & Outputs](https://mlsanity.com/paper/2107.14795/) (2021)
- [Interaction Networks for Learning about Objects, Relations and Physics](https://mlsanity.com/paper/1612.00222/) (2016)
- [Deep learning for universal linear embeddings of nonlinear dynamics](https://mlsanity.com/paper/1712.09707/) (2017)
- [MACE: Higher Order Equivariant Message Passing Neural Networks for Fast and Accurate Force Fields](https://mlsanity.com/paper/2206.07697/) (2022)
- [Machine learning accelerated computational fluid dynamics](https://mlsanity.com/paper/2102.01010/) (2021)
- [Symbolic Discovery of Optimization Algorithms](https://mlsanity.com/paper/2302.06675/) (2023)
- [Neural Operator: Learning Maps Between Function Spaces](https://mlsanity.com/paper/2108.08481/) (2021)

## Built on by (most cited first, 27 of 27)

- [Vision-LSTM: xLSTM as Generic Vision Backbone](https://mlsanity.com/paper/2406.04303/) (2024)
- [NeuralDEM -- Real-time Simulation of Industrial Particulate Flows](https://mlsanity.com/paper/2411.09678/) (2024)
- [Diffusion-Based Hierarchical Graph Neural Networks for Simulating Nonlinear Solid Mechanics](https://mlsanity.com/paper/2506.06045/) (2025)
- [OmniArch: Building Foundation Model For Scientific Computing](https://mlsanity.com/paper/2402.16014/) (2024)
- [GyroSwin: 5D Surrogates for Gyrokinetic Plasma Turbulence Simulations](https://mlsanity.com/paper/2510.07314/) (2025)
- [Neural Green's Functions](https://mlsanity.com/paper/2511.01924/) (2025)
- [Pretraining a Neural Operator in Lower Dimensions](https://mlsanity.com/paper/2407.17616/) (2024)
- [Micrometer: Micromechanics Transformer for Predicting Mechanical Responses of Heterogeneous Materials](https://mlsanity.com/paper/2410.05281/) (2024)
- [DiffusionRollout: Uncertainty-Aware Rollout Planning in Long-Horizon PDE Solving](https://mlsanity.com/paper/2602.13616/) (2026)
- [Learning Lagrangian Interaction Dynamics with Sampling-Based Model Order Reduction](https://mlsanity.com/paper/2407.03925/) (2024)
- [Implicit factorized transformer approach to fast prediction of turbulent channel flows](https://mlsanity.com/paper/2412.18840/) (2024)
- [A Neural Operator Emulator for Coastal and Riverine Shallow Water Dynamics](https://mlsanity.com/paper/2502.14782/) (2025)
- [PAINT: Parallel-in-time Neural Twins for Dynamical System Reconstruction](https://mlsanity.com/paper/2510.16004/) (2025)
- [Modeling Language as a Sequence of Thoughts](https://mlsanity.com/paper/2512.25026/) (2025)
- [Di-BiLPS: Denoising induced Bidirectional Latent-PDE-Solver under Sparse Observations](https://mlsanity.com/paper/2605.13790/) (2026)
- [Neuro-Symbolic AI for Analytical Solutions of Differential Equations](https://mlsanity.com/paper/2502.01476/) (2025)
- [BSA: Ball Sparse Attention for Large-scale Geometries](https://mlsanity.com/paper/2506.12541/) (2025)
- [Adaptive Physics Transformer with Fused Global-Local Attention for Subsurface Energy Systems](https://mlsanity.com/paper/2602.11208/) (2026)
- [Zero-shot generalization of transformer neural operators to larger domains](https://mlsanity.com/paper/2606.14597/) (2026)
- [Generalized Neural Operator for Parametric and Boundary-Value Problems](https://mlsanity.com/paper/2607.21932/) (2026)
- [DamFormer: Generalizing Morphologies in Dam Break Simulations Using Transformer Model](https://mlsanity.com/paper/2410.18998/) (2024)
- [Physics Consistency and Latent Dynamics in Spatiotemporal Physics Field Generation](https://mlsanity.com/paper/2505.11578/) (2025)
- [Towards Near-Real-Time Telemetry-Aware Routing with Neural Routing Algorithms](https://mlsanity.com/paper/2604.02927/) (2026)
- [LEIA: Learned Environment for Interactive Architected Materials](https://mlsanity.com/paper/2605.28368/) (2026)
- [Hybrid Lagrangian-Eulerian Model for Lagrangian Fluid Simulation](https://mlsanity.com/paper/2608.01164/) (2026)
- [ONE CYLinder: A Benchmark for Graph-Based Surrogate Modeling of Unsteady Bluff-Body Flows](https://mlsanity.com/paper/2609.08947/) (2026)
- [AutoPDEBench: Benchmarking LLM Auto-Research for Neural PDE Solver Design](https://mlsanity.com/paper/2609.32245/) (2026)

## Abstract (arXiv)

Neural operators, serving as physics surrogate models, have recently gained increased interest. With ever increasing problem complexity, the natural question arises: what is an efficient way to scale neural operators to larger and more complex simulations - most importantly by taking into account different types of simulation datasets. This is of special interest since, akin to their numerical counterparts, different techniques are used across applications, even if the underlying dynamics of the systems are similar. Whereas the flexibility of transformers has enabled unified architectures across domains, neural operators mostly follow a problem specific design, where GNNs are commonly used for Lagrangian simulations and grid-based models predominate Eulerian simulations. We introduce Universal Physics Transformers (UPTs), an efficient and unified learning paradigm for a wide range of spatio-temporal problems. UPTs operate without grid- or particle-based latent structures, enabling flexibility and scalability across meshes and particles. UPTs efficiently propagate dynamics in the latent space, emphasized by inverse encoding and decoding techniques. Finally, UPTs allow for queries of the latent space representation at any point in space-time. We demonstrate diverse applicability and efficacy of UPTs in mesh-based fluid simulations, and steady-state Reynolds averaged Navier-Stokes simulations, and Lagrangian-based dynamics.