# PROSE-FD: A Multimodal PDE Foundation Model for Learning Multiple Operators for Forecasting Fluid Dynamics

arXiv: https://arxiv.org/abs/2409.09811 · HTML: https://mlsanity.com/paper/2409.09811/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 50 references that are in the map.
- First posted on arXiv 15 Sep 2024.

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operators](https://mlsanity.com/paper/2202.11214/) (2022)
- [MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies](https://mlsanity.com/paper/2404.06395/) (2024)
- [VideoBERT: A Joint Model for Video and Language Representation Learning](https://mlsanity.com/paper/1904.01766/) (2019)
- [AI Choreographer: Music Conditioned 3D Dance Generation with AIST++](https://mlsanity.com/paper/2101.08779/) (2021)
- [Fourier Neural Operator with Learned Deformations for PDEs on General Geometries](https://mlsanity.com/paper/2207.05209/) (2022)
- [PDEBENCH: An Extensive Benchmark for Scientific Machine Learning](https://mlsanity.com/paper/2210.07182/) (2022)
- [Multimodal Learning with Transformers: A Survey](https://mlsanity.com/paper/2206.06488/) (2022)
- [Axial Attention in Multidimensional Transformers](https://mlsanity.com/paper/1912.12180/) (2019)
- [Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformers](https://mlsanity.com/paper/2111.13587/) (2021)
- [End-to-end symbolic regression with transformers](https://mlsanity.com/paper/2204.10532/) (2022)
- [Towards Multi-spatiotemporal-scale Generalized PDE Modeling](https://mlsanity.com/paper/2209.15616/) (2022)
- [Dual Attention Networks for Multimodal Reasoning and Matching](https://mlsanity.com/paper/1611.00471/) (2016)

## Built on by (most cited first, 30 of 34)

- [VICON: Vision In-Context Operator Networks for Multi-Physics Fluid Dynamics Prediction](https://mlsanity.com/paper/2411.16063/) (2024)
- [BCAT: A Block Causal Transformer for PDE Foundation Models for Fluid Dynamics](https://mlsanity.com/paper/2501.18972/) (2025)
- [Neural Scaling Laws of Deep ReLU and Deep Operator Network: A Theoretical Study](https://mlsanity.com/paper/2410.00357/) (2024)
- [MORPH: PDE Foundation Models with Arbitrary Data Modality](https://mlsanity.com/paper/2509.21670/) (2025)
- [DeepONet as a Multi-Operator Extrapolation Model: Distributed Pretraining with Physics-Informed Fine-Tuning](https://mlsanity.com/paper/2411.07239/) (2024)
- [A Multimodal PDE Foundation Model for Prediction and Scientific Text Descriptions](https://mlsanity.com/paper/2502.06026/) (2025)
- [A Deep Learning Framework for Multi-Operator Learning: Architectures and Approximation Theory](https://mlsanity.com/paper/2510.25379/) (2025)
- [Time-Series Forecasting, Knowledge Distillation, and Refinement within a Multimodal PDE Foundation Model](https://mlsanity.com/paper/2409.11609/) (2024)
- [Foundation Models for Scientific Discovery: From Paradigm Enhancement to Paradigm Transition](https://mlsanity.com/paper/2510.15280/) (2025)
- [Cauchy Random Features for Operator Learning in Sobolev Space](https://mlsanity.com/paper/2503.00300/) (2025)
- [Physics-informed Temporal Alignment for Auto-regressive PDE Foundation Models](https://mlsanity.com/paper/2505.10930/) (2025)
- [CompNO: A Novel Foundation Model approach for solving Partial Differential Equations](https://mlsanity.com/paper/2601.07384/) (2026)
- [OpInf-LLM: Parametric PDE Solving with LLMs via Operator Inference](https://mlsanity.com/paper/2602.01493/) (2026)
- [Flow marching for a generative PDE foundation model](https://mlsanity.com/paper/2509.18611/) (2025)
- [Multiple Neural Operators Achieve Near-Optimal Rates for Multi-Task Learning](https://mlsanity.com/paper/2605.22724/) (2026)
- [Emergent Transfer of a Physics Foundation Model from Simulation to Laboratory Turbulence](https://mlsanity.com/paper/2606.01470/) (2026)
- [Continuum Transformers Perform In-Context Learning by Operator Gradient Descent](https://mlsanity.com/paper/2505.17838/) (2025)
- [Latent Generative Solvers for Generalizable Long-Term Physics Simulation](https://mlsanity.com/paper/2602.11229/) (2026)
- [Graph In-Context Operator Networks for Generalizable Spatiotemporal Prediction](https://mlsanity.com/paper/2603.12725/) (2026)
- [Generalization Bounds and Statistical Guarantees for Multi-Task and Multiple Operator Learning with MNO Networks](https://mlsanity.com/paper/2604.01961/) (2026)
- [Convolutional-neural-operator-based transfer learning for solving PDEs](https://mlsanity.com/paper/2512.17969/) (2025)
- [SPUS: A Lightweight and Parameter-Efficient Foundation Model for PDEs](https://mlsanity.com/paper/2510.01370/) (2025)
- [OmniField: Conditioned Neural Fields for Robust Multimodal Spatiotemporal Learning](https://mlsanity.com/paper/2511.02205/) (2025)
- [MVNN: A Measure-Valued Neural Network for Learning McKean-Vlasov Dynamics from Particle Data](https://mlsanity.com/paper/2604.00333/) (2026)
- [A Multimodal Vision Transformer-based Modeling Framework for Prediction of Fluid Flows in Energy Systems](https://mlsanity.com/paper/2604.02483/) (2026)
- [Harnessing AI for Inverse Partial Differential Equation Problems: Past, Present, and Prospects](https://mlsanity.com/paper/2605.16966/) (2026)
- [FMint-SDE: A Multimodal Foundation Model for Accelerating Numerical Simulation of SDEs via Error Correction](https://mlsanity.com/paper/2510.27173/) (2025)
- [In-Context Multi-Operator Learning with DeepOSets](https://mlsanity.com/paper/2512.16074/) (2025)
- [Compositional Neural Operators for Multi-Dimensional Fluid Dynamics](https://mlsanity.com/paper/2605.11691/) (2026)
- [Kastor: An efficient fine-tuning strategy for generative emulation of PDE simulations](https://mlsanity.com/paper/2608.06107/) (2026)

## Abstract (arXiv)

We propose PROSE-FD, a zero-shot multimodal PDE foundational model for simultaneous prediction of heterogeneous two-dimensional physical systems related to distinct fluid dynamics settings. These systems include shallow water equations and the Navier-Stokes equations with incompressible and compressible flow, regular and complex geometries, and different buoyancy settings. This work presents a new transformer-based multi-operator learning approach that fuses symbolic information to perform operator-based data prediction, i.e. non-autoregressive. By incorporating multiple modalities in the inputs, the PDE foundation model builds in a pathway for including mathematical descriptions of the physical behavior. We pre-train our foundation model on 6 parametric families of equations collected from 13 datasets, including over 60K trajectories. Our model outperforms popular operator learning, computer vision, and multi-physics models, in benchmark forward prediction tasks. We test our architecture choices with ablation studies.