# Towards a Physics Foundation Model

arXiv: https://arxiv.org/abs/2509.13805 · HTML: https://mlsanity.com/paper/2509.13805/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 55 references that are in the map.
- First posted on arXiv 17 Sep 2025.

## Builds on (most cited first, 30 of 38)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [MaskGIT: Masked Generative Image Transformer](https://mlsanity.com/paper/2202.04200/) (2022)
- [Deep Networks with Stochastic Depth](https://mlsanity.com/paper/1603.09382/) (2016)
- [Physics-Informed Neural Operator for Learning Partial Differential Equations](https://mlsanity.com/paper/2111.03794/) (2021)
- [Large Language Models: A Survey](https://mlsanity.com/paper/2402.06196/) (2024)
- [ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction](https://mlsanity.com/paper/2010.09885/) (2020)
- [ClimaX: A foundation model for weather and climate](https://mlsanity.com/paper/2301.10343/) (2023)
- [Many-Shot In-Context Learning](https://mlsanity.com/paper/2404.11018/) (2024)
- [Poseidon: Efficient Foundation Models for PDEs](https://mlsanity.com/paper/2405.19101/) (2024)
- [DPOT: Auto-Regressive Denoising Operator Transformer for Large-Scale PDE Pre-Training](https://mlsanity.com/paper/2403.03542/) (2024)
- [The Well: a Large-Scale Collection of Diverse Physics Simulations for Machine Learning](https://mlsanity.com/paper/2412.00568/) (2024)
- [Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer Behavior](https://mlsanity.com/paper/2306.00258/) (2023)
- [Transfer learning enhanced physics informed neural network for phase-field modeling of fracture](https://mlsanity.com/paper/1907.02531/) (2019)
- [Physics-Informed Deep Neural Operator Networks](https://mlsanity.com/paper/2207.05748/) (2022)
- [Deep transfer operator learning for partial differential equations under conditional shift](https://mlsanity.com/paper/2204.09810/) (2022)
- [Hypernetwork-based Meta-Learning for Low-Rank Physics-Informed Neural Networks](https://mlsanity.com/paper/2310.09528/) (2023)
- [AstroLLaMA: Towards Specialized Foundation Models in Astronomy](https://mlsanity.com/paper/2309.06126/) (2023)
- [A Library for Learning Neural Operators](https://mlsanity.com/paper/2412.10354/) (2024)
- [A survey of the Vision Transformers and their CNN-Transformer based Variants](https://mlsanity.com/paper/2305.09880/) (2023)

## Built on by (most cited first, 8 of 8)

- [Cognitive Mismatch in Multimodal Large Language Models for Discrete Symbol Understanding](https://mlsanity.com/paper/2603.18472/) (2026)
- [OpInf-LLM: Parametric PDE Solving with LLMs via Operator Inference](https://mlsanity.com/paper/2602.01493/) (2026)
- [OmniFysics: Towards Physical Intelligence Evolution via Omni-Modal Signal Processing and Network Optimization](https://mlsanity.com/paper/2602.07064/) (2026)
- [Tadpole: Autoencoders as Foundation Models for 3D PDEs with Online Learning](https://mlsanity.com/paper/2605.15284/) (2026)
- [Pretrained Video Models as Differentiable Physics Simulators for Urban Wind Flows](https://mlsanity.com/paper/2603.21210/) (2026)
- [Do Physics Foundation Models Learn Generalizable Physics? A Bias-Aware Benchmark Across Physical Regimes and Distribution Shifts](https://mlsanity.com/paper/2605.29283/) (2026)
- [Stability of Measure-to-Measure Transformers on Sub-Gaussian Data](https://mlsanity.com/paper/2610.07717/) (2026)
- [One-Way Coupling via Patchwise Adaptive Normalization in Transformer Surrogates](https://mlsanity.com/paper/2610.14687/) (2026)

## Abstract (arXiv)

Foundation models have revolutionized natural language processing through a ``train once, deploy anywhere'' paradigm, where a single pre-trained model adapts to countless downstream tasks without retraining. Access to a Physics Foundation Model (PFM) would be transformative - democratizing access to high-fidelity simulations, accelerating scientific discovery, and eliminating the need for specialized solver development. Yet current physics-aware machine learning approaches remain fundamentally limited to single, narrow domains and require retraining for each new system. We present the General Physics Transformer (GPhyT), trained on 1.8 TB of diverse simulation data, that demonstrates foundation model capabilities are achievable for physics. Our key insight is that transformers can learn to infer governing dynamics from context, enabling a single model to simulate fluid-solid interactions, shock waves, thermal convection, and multi-phase dynamics without being told the underlying equations. GPhyT achieves three critical breakthroughs: (1) superior performance across multiple physics domains, outperforming specialized architectures by more than 7x, (2) plausible zero-shot generalization to entirely unseen physical systems through in-context learning, and (3) more stable long-term predictions through long-horizon rollouts. By establishing that a single model can learn generalizable physical principles from data alone, this work opens the path toward a universal PFM that could transform computational science and engineering.