# Defining Foundation Models for Computational Science: A Call for Clarity and Rigor

arXiv: https://arxiv.org/abs/2505.22904 · HTML: https://mlsanity.com/paper/2505.22904/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 61 references that are in the map.
- First posted on arXiv 28 May 2025.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operators](https://mlsanity.com/paper/2202.11214/) (2022)
- [Weak SINDy For Partial Differential Equations](https://mlsanity.com/paper/2007.02848/) (2020)
- [Foundation Models for Generalist Geospatial Artificial Intelligence](https://mlsanity.com/paper/2310.18660/) (2023)
- [The Well: a Large-Scale Collection of Diverse Physics Simulations for Machine Learning](https://mlsanity.com/paper/2412.00568/) (2024)
- [Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer Behavior](https://mlsanity.com/paper/2306.00258/) (2023)
- [A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4](https://mlsanity.com/paper/2310.12321/) (2023)
- [A fast and accurate physics-informed neural network reduced order model with shallow masked autoencoder](https://mlsanity.com/paper/2009.11990/) (2020)
- [Generalized Multiscale Finite Element Methods (GMsFEM)](https://mlsanity.com/paper/1301.2866/) (2013)
- [Component-wise reduced order model lattice-type structure design](https://mlsanity.com/paper/2010.10770/) (2020)
- [Train Small, Model Big: Scalable Physics Simulators via Reduced Order Modeling and Domain Decomposition](https://mlsanity.com/paper/2401.10245/) (2023)
- [Tokenization for Molecular Foundation Models](https://mlsanity.com/paper/2409.15370/) (2024)
- [The role of interface boundary conditions and sampling strategies for Schwarz-based coupling of projection-based reduced order models](https://mlsanity.com/paper/2410.04668/) (2024)
- [Geometry Matters: Benchmarking Scientific ML Approaches for Flow Prediction around Complex Geometries](https://mlsanity.com/paper/2501.01453/) (2024)

## Built on by (most cited first, 17 of 17)

- [Towards a Physics Foundation Model](https://mlsanity.com/paper/2509.13805/) (2025)
- [Structure-Preserving Learning Improves Geometry Generalization in Neural PDEs](https://mlsanity.com/paper/2602.02788/) (2026)
- [ATHENA: Agentic Team for Hierarchical Evolutionary Numerical Algorithms](https://mlsanity.com/paper/2512.03476/) (2025)
- [CompNO: A Novel Foundation Model approach for solving Partial Differential Equations](https://mlsanity.com/paper/2601.07384/) (2026)
- [Hybrid coupling with operator inference and the overlapping Schwarz alternating method](https://mlsanity.com/paper/2511.20687/) (2025)
- [SLE-FNO: Single-Layer Extensions for Task-Agnostic Continual Learning in Fourier Neural Operators](https://mlsanity.com/paper/2603.20410/) (2026)
- [A Mechanistic Analysis of Transformers for Dynamical Systems](https://mlsanity.com/paper/2512.21113/) (2025)
- [Latent Space Element Method](https://mlsanity.com/paper/2601.01741/) (2026)
- [Learning Mesh-Free Discrete Differential Operators with Self-Supervised Graph Neural Networks](https://mlsanity.com/paper/2603.24641/) (2026)
- [Toward Generalizable Graph Learning for 3D Engineering AI: Explainable Workflows for CAE Mode Shape Classification and CFD Field Prediction](https://mlsanity.com/paper/2604.07781/) (2026)
- [A Hybridizable Neural Time Integrator for Stable Autoregressive Forecasting](https://mlsanity.com/paper/2604.21101/) (2026)
- [Replay-Based Continual Learning for Physics-Informed Neural Operators](https://mlsanity.com/paper/2605.04832/) (2026)
- [Compositional Neural Operators for Multi-Dimensional Fluid Dynamics](https://mlsanity.com/paper/2605.11691/) (2026)
- [From inverse problems to neural operators: prediction, mechanism, and generalization of data-driven models](https://mlsanity.com/paper/2606.08956/) (2026)
- [Advances in Scientific Machine Learning for Coupled Fluid Flow and Transport](https://mlsanity.com/paper/2606.19562/) (2026)
- [Learning Spectrally Optimised Mesh-Free Discretisations](https://mlsanity.com/paper/2609.02833/) (2026)
- [Into the danger zone: stable extrapolation in high-dimensional function and operator learning](https://mlsanity.com/paper/2609.36709/) (2026)

## Abstract (arXiv)

The widespread success of foundation models in natural language processing and computer vision has inspired researchers to extend the concept to scientific machine learning and computational science. However, this position paper argues that as the term "foundation model" is an evolving concept, its application in computational science is increasingly used without a universally accepted definition, potentially creating confusion and diluting its precise scientific meaning. In this paper, we address this gap by proposing a formal definition of foundation models in computational science, grounded in the core values of generality, reusability, and scalability. We articulate a set of essential and desirable characteristics that such models must exhibit, drawing parallels with traditional foundational methods, like the finite element and finite volume methods. Furthermore, we introduce the Data-Driven Finite Element Method (DD-FEM), a framework that fuses the modular structure of classical FEM with the representational power of data-driven learning. We demonstrate how DD-FEM addresses many of the key challenges in realizing foundation models for computational science, including scalability, adaptability, and physics consistency. By bridging traditional numerical methods with modern AI paradigms, this work provides a rigorous foundation for evaluating and developing novel approaches toward future foundation models in computational science.