# Pretraining Codomain Attention Neural Operators for Solving Multiphysics PDEs

arXiv: https://arxiv.org/abs/2403.12553 · HTML: https://mlsanity.com/paper/2403.12553/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 45 references that are in the map.
- First posted on arXiv 19 Mar 2024.

## Builds on (most cited first, 30 of 31)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [Neural Operator: Graph Kernel Network for Partial Differential Equations](https://mlsanity.com/paper/2003.03485/) (2020)
- [Interaction Networks for Learning about Objects, Relations and Physics](https://mlsanity.com/paper/1612.00222/) (2016)
- [Neural Operator: Learning Maps Between Function Spaces](https://mlsanity.com/paper/2108.08481/) (2021)
- [PDEBENCH: An Extensive Benchmark for Scientific Machine Learning](https://mlsanity.com/paper/2210.07182/) (2022)
- [Geometry-Informed Neural Operator for Large-Scale 3D PDEs](https://mlsanity.com/paper/2309.00583/) (2023)
- [Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformers](https://mlsanity.com/paper/2111.13587/) (2021)
- [Spherical Fourier Neural Operators: Learning Stable Dynamics on the Sphere](https://mlsanity.com/paper/2306.03838/) (2023)
- [Neural Operators for Accelerating Scientific Simulations and Design](https://mlsanity.com/paper/2309.15325/) (2023)
- [StyleSwin: Transformer-based GAN for High-resolution Image Generation](https://mlsanity.com/paper/2112.10762/) (2021)
- [DPOT: Auto-Regressive Denoising Operator Transformer for Large-Scale PDE Pre-Training](https://mlsanity.com/paper/2403.03542/) (2024)
- [Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer Behavior](https://mlsanity.com/paper/2306.00258/) (2023)
- [Scalable Transformer for PDE Surrogate Modeling](https://mlsanity.com/paper/2305.17560/) (2023)
- [Multiple Physics Pretraining for Physical Surrogate Models](https://mlsanity.com/paper/2310.02994/) (2023)
- [Multi-Grid Tensorized Fourier Neural Operator for High-Resolution PDEs](https://mlsanity.com/paper/2310.00120/) (2023)
- [Real-time high-resolution CO2 geological storage prediction using nested Fourier neural operators](https://mlsanity.com/paper/2210.17051/) (2022)
- [Learning Neural PDE Solvers with Parameter-Guided Channel Attention](https://mlsanity.com/paper/2304.14118/) (2023)

## Built on by (most cited first, 30 of 59)

- [The Well: a Large-Scale Collection of Diverse Physics Simulations for Machine Learning](https://mlsanity.com/paper/2412.00568/) (2024)
- [Continuum Attention for Neural Operators](https://mlsanity.com/paper/2406.06486/) (2024)
- [Walrus: A Cross-Domain Foundation Model for Continuum Dynamics](https://mlsanity.com/paper/2511.15684/) (2025)
- [Zero-shot forecasting of chaotic systems](https://mlsanity.com/paper/2409.15771/) (2024)
- [FMint: Bridging Human Designed and Data Pretrained Models for Differential Equation Foundation Model](https://mlsanity.com/paper/2404.14688/) (2024)
- [Context parroting: A simple but tough-to-beat baseline for foundation models in scientific machine learning](https://mlsanity.com/paper/2505.11349/) (2025)
- [Masked Autoencoders are PDE Learners](https://mlsanity.com/paper/2403.17728/) (2024)
- [Strategies for Pretraining Neural Operators](https://mlsanity.com/paper/2406.08473/) (2024)
- [LeMON: Learning to Learn Multi-Operator Networks](https://mlsanity.com/paper/2408.16168/) (2024)
- [Principled Approaches for Extending Neural Architectures to Function Spaces for Operator Learning](https://mlsanity.com/paper/2506.10973/) (2025)
- [Lang-PINN: From Language to Physics-Informed Neural Networks via a Multi-Agent Framework](https://mlsanity.com/paper/2510.05158/) (2025)
- [Operator Learning at Machine Precision](https://mlsanity.com/paper/2511.19980/) (2025)
- [An Advanced Physics-Informed Neural Operator for Comprehensive Design Optimization of Highly-Nonlinear Systems: An Aerospace Composites Processing Case Study](https://mlsanity.com/paper/2406.14715/) (2024)
- [Foundation Models for Scientific Discovery: From Paradigm Enhancement to Paradigm Transition](https://mlsanity.com/paper/2510.15280/) (2025)
- [EddyFormer: Accelerated Neural Simulations of Three-Dimensional Turbulence at Scale](https://mlsanity.com/paper/2510.24173/) (2025)
- [CompNO: A Novel Foundation Model approach for solving Partial Differential Equations](https://mlsanity.com/paper/2601.07384/) (2026)
- [Data-Efficient Inference of Neural Fluid Fields via SciML Foundation Model](https://mlsanity.com/paper/2412.13897/) (2024)
- [Residuals Are Not Enough: Limits of Physics-Informed Pre-Training for Scientific Foundation Models](https://mlsanity.com/paper/2503.19081/) (2025)
- [Emergent Transfer of a Physics Foundation Model from Simulation to Laboratory Turbulence](https://mlsanity.com/paper/2606.01470/) (2026)
- [Liquid Fourier Latent Dynamics Networks for fast GPU-based numerical simulations in computational cardiology](https://mlsanity.com/paper/2408.09818/) (2024)
- [Group Downsampling with Equivariant Anti-aliasing](https://mlsanity.com/paper/2504.17258/) (2025)
- [F-Adapter: Frequency-Adaptive Parameter-Efficient Fine-Tuning in Scientific Machine Learning](https://mlsanity.com/paper/2509.23173/) (2025)
- [Discovering Physical Directions in Weight Space: Composing Neural PDE Experts](https://mlsanity.com/paper/2605.14546/) (2026)
- [Learning Lagrangian Interaction Dynamics with Sampling-Based Model Order Reduction](https://mlsanity.com/paper/2407.03925/) (2024)
- [M2M: Learning controllable Multi of experts and multi-scale operators are the Partial Differential Equations need](https://mlsanity.com/paper/2410.11617/) (2024)
- [Single vs. Multiple Branches in DeepONet and S-DeepONet: Network Architecture Follows Coupling in Multiphysics Systems](https://mlsanity.com/paper/2507.03660/) (2025)
- [DRIFT-Net: A Spectral--Coupled Neural Operator for PDEs Learning](https://mlsanity.com/paper/2509.24868/) (2025)
- [GenCP: Towards Generative Modeling Paradigm of Coupled Physics](https://mlsanity.com/paper/2601.19541/) (2026)
- [Transformers for dynamical systems learn transfer operators in-context](https://mlsanity.com/paper/2602.18679/) (2026)
- [Learning Physical Operators using Neural Operators](https://mlsanity.com/paper/2602.23113/) (2026)

## Abstract (arXiv)

Existing neural operator architectures face challenges when solving multiphysics problems with coupled partial differential equations (PDEs) due to complex geometries, interactions between physical variables, and the limited amounts of high-resolution training data. To address these issues, we propose Codomain Attention Neural Operator (CoDA-NO), which tokenizes functions along the codomain or channel space, enabling self-supervised learning or pretraining of multiple PDE systems. Specifically, we extend positional encoding, self-attention, and normalization layers to function spaces. CoDA-NO can learn representations of different PDE systems with a single model. We evaluate CoDA-NO's potential as a backbone for learning multiphysics PDEs over multiple systems by considering few-shot learning settings. On complex downstream tasks with limited data, such as fluid flow simulations, fluid-structure interactions, and Rayleigh-B'enard convection, we found CoDA-NO to outperform existing methods by over 36%.