# Walrus: A Cross-Domain Foundation Model for Continuum Dynamics

arXiv: https://arxiv.org/abs/2511.15684 · HTML: https://mlsanity.com/paper/2511.15684/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 60 of its 89 references that are in the map.
- First posted on arXiv 19 Nov 2025.

## Builds on (most cited first, 30 of 60)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [Alias-Free Generative Adversarial Networks](https://mlsanity.com/paper/2106.12423/) (2021)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [Scaling Vision Transformers to 22 Billion Parameters](https://mlsanity.com/paper/2302.05442/) (2023)
- [DGM: A deep learning algorithm for solving partial differential equations](https://mlsanity.com/paper/1708.07469/) (2017)
- [The Deep Ritz method: A deep learning-based numerical algorithm for solving variational problems](https://mlsanity.com/paper/1710.00211/) (2017)
- [Solving high-dimensional partial differential equations using deep learning](https://mlsanity.com/paper/1707.02568/) (2017)
- [Physics-Informed Neural Operator for Learning Partial Differential Equations](https://mlsanity.com/paper/2111.03794/) (2021)
- [The frontier of simulation-based inference](https://mlsanity.com/paper/1911.01429/) (2019)
- [Machine learning accelerated computational fluid dynamics](https://mlsanity.com/paper/2102.01010/) (2021)
- [PDEBENCH: An Extensive Benchmark for Scientific Machine Learning](https://mlsanity.com/paper/2210.07182/) (2022)
- [Axial Attention in Multidimensional Transformers](https://mlsanity.com/paper/1912.12180/) (2019)
- [Choose a Transformer: Fourier or Galerkin](https://mlsanity.com/paper/2105.14995/) (2021)
- [ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction](https://mlsanity.com/paper/2010.09885/) (2020)
- [ClimaX: A foundation model for weather and climate](https://mlsanity.com/paper/2301.10343/) (2023)
- [Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action](https://mlsanity.com/paper/2312.17172/) (2023)
- [GraphCast: Learning skillful medium-range global weather forecasting](https://mlsanity.com/paper/2212.12794/) (2022)
- [Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformers](https://mlsanity.com/paper/2111.13587/) (2021)
- [Convolutional Neural Operators for robust and accurate learning of PDEs](https://mlsanity.com/paper/2302.01178/) (2023)
- [Solver-in-the-Loop: Learning from Differentiable Physics to Interact with Iterative PDE-Solvers](https://mlsanity.com/paper/2007.00016/) (2020)

## Built on by (most cited first, 30 of 36)

- [Representation Learning for Spatiotemporal Physical Systems](https://mlsanity.com/paper/2603.13227/) (2026)
- [Test-time Generalization for Physics through Neural Operator Splitting](https://mlsanity.com/paper/2602.00884/) (2026)
- [Structure-Preserving Learning Improves Geometry Generalization in Neural PDEs](https://mlsanity.com/paper/2602.02788/) (2026)
- [Emergent Transfer of a Physics Foundation Model from Simulation to Laboratory Turbulence](https://mlsanity.com/paper/2606.01470/) (2026)
- [Physics-informed fine-tuning of foundation models for partial differential equations](https://mlsanity.com/paper/2603.15431/) (2026)
- [Discovering Physical Directions in Weight Space: Composing Neural PDE Experts](https://mlsanity.com/paper/2605.14546/) (2026)
- [Tadpole: Autoencoders as Foundation Models for 3D PDEs with Online Learning](https://mlsanity.com/paper/2605.15284/) (2026)
- [Sparse probes and murky physics: a case study of interpretability challenges in a foundation model for continuum dynamics](https://mlsanity.com/paper/2606.11657/) (2026)
- [PDE foundation models are skillful AI weather emulators for the Martian atmosphere](https://mlsanity.com/paper/2602.15004/) (2026)
- [Transformers for dynamical systems learn transfer operators in-context](https://mlsanity.com/paper/2602.18679/) (2026)
- [Autoregressive long-horizon prediction of plasma edge dynamics](https://mlsanity.com/paper/2512.23884/) (2025)
- [Towards a Foundation Model for the Martian Atmosphere](https://mlsanity.com/paper/2605.28851/) (2026)
- [Pre-Training for Simulation-Based Science: A Study on Jet Foundation Model Training Objectives](https://mlsanity.com/paper/2606.14870/) (2026)
- [NEXUS: Neural Energy Fields for Physically Consistent Contact-Rich 3D Object Dynamics](https://mlsanity.com/paper/2606.15015/) (2026)
- [Preconditioned Physics-Informed Neural Operator Training](https://mlsanity.com/paper/2609.36216/) (2026)
- [Sim+Real: Joint Simulation-Experiment Training Improves Balanced Prediction in Physical Systems](https://mlsanity.com/paper/2610.01974/) (2026)
- [Deterministic and probabilistic neural surrogates of global hybrid-Vlasov simulations](https://mlsanity.com/paper/2601.12614/) (2026)
- [Replay-Based Continual Learning for Physics-Informed Neural Operators](https://mlsanity.com/paper/2605.04832/) (2026)
- [Autoregressive One-Step Generative Modeling for Dynamical System Forecasting](https://mlsanity.com/paper/2605.05540/) (2026)
- [Eradicating Negative Transfer in Multi-Physics Foundation Models via Sparse Mixture-of-Experts Routing](https://mlsanity.com/paper/2605.15179/) (2026)
- [Breakeven complexity: A new perspective on neural partial differential equation solvers](https://mlsanity.com/paper/2605.15399/) (2026)
- [AOT-POT: Adaptive Operator Transformation for Large-Scale PDE Pre-training](https://mlsanity.com/paper/2605.15793/) (2026)
- [Small Models, Strong Priors: Architectural Inductive Bias for Parameter-Efficient Neural PDE Solvers](https://mlsanity.com/paper/2605.25949/) (2026)
- [Towards Data-Efficient Cross-Device Generalization of Grad-Shafranov Equilibria via Transfer Learning Neural Operator](https://mlsanity.com/paper/2606.15512/) (2026)
- [Distillation of Foundation Models for Time-dependent PDEs](https://mlsanity.com/paper/2608.11937/) (2026)
- [A survey detection channel overrides the pixels in an astronomical foundation model, and biases tomographic mean redshifts](https://mlsanity.com/paper/2608.23626/) (2026)
- [From Numerical Simulators of PDEs to Neural Emulators and Back](https://mlsanity.com/paper/2608.24547/) (2026)
- [Beyond Compression: Training Latent Representations for Stable Long-Horizon Rollout in Neural Surrogate Solvers](https://mlsanity.com/paper/2609.30198/) (2026)
- [PDE-JEPA: Predictive Representation Learning of Latent Dynamics Modeling for Parametric PDEs](https://mlsanity.com/paper/2609.34715/) (2026)
- [Geometry Meets Physics: Data-Efficient Pre-Training for Unstructured Neural PDE Solvers](https://mlsanity.com/paper/2610.03363/) (2026)

## Abstract (arXiv)

Foundation models have transformed machine learning for language and vision, but achieving comparable impact in physical simulation remains a challenge. Data heterogeneity and unstable long-term dynamics inhibit learning from sufficiently diverse dynamics, while varying resolutions and dimensionalities challenge efficient training on modern hardware. Through empirical and theoretical analysis, we incorporate new approaches to mitigate these obstacles, including a harmonic-analysis-based stabilization method, load-balanced distributed 2D and 3D training strategies, and compute-adaptive tokenization. Using these tools, we develop Walrus, a transformer-based foundation model developed primarily for fluid-like continuum dynamics. Walrus is pretrained on nineteen diverse scenarios spanning astrophysics, geoscience, rheology, plasma physics, acoustics, and classical fluids. Experiments show that Walrus outperforms prior foundation models on both short and long term prediction horizons on downstream tasks and across the breadth of pretraining data, while ablation studies confirm the value of our contributions to forecast stability, training throughput, and transfer performance over conventional approaches. Code and weights are released for community use.