mlsanityData to 14 Oct 2026

Pretraining Codomain Attention Neural Operators for Solving Multiphysics PDEs

arXiv 2403.12553 ↗First posted 19 Mar 2024cs.LGcited by 59

Not on the map. Why?

Abstract (arXiv)

Existing neural operator architectures face challenges when solving multiphysics problems with coupled partial differential equations (PDEs) due to complex geometries, interactions between physical variables, and the limited amounts of high-resolution training data. To address these issues, we propose Codomain Attention Neural Operator (CoDA-NO), which tokenizes functions along the codomain or channel space, enabling self-supervised learning or pretraining of multiple PDE systems. Specifically, we extend positional encoding, self-attention, and normalization layers to function spaces. CoDA-NO can learn representations of different PDE systems with a single model. We evaluate CoDA-NO's potential as a backbone for learning multiphysics PDEs over multiple systems by considering few-shot learning settings. On complex downstream tasks with limited data, such as fluid flow simulations, fluid-structure interactions, and Rayleigh-B'enard convection, we found CoDA-NO to outperform existing methods by over 36%.

Key facts. Cited by 59 papers in the map (data through 14 Oct 2026). Builds on 31 of its 45 references that are in the map. First posted on arXiv 19 Mar 2024.

Builds on

31 of 45 references.

References · 31

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  7. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  10. 2016Layer Normalizationcited by 5,669
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  12. 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,908
19 more
  1. 2018Group Normalizationcited by 1,524
  2. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,198
  3. 2020Neural Operator: Graph Kernel Network for Partial Differential Equationscited by 734
  4. 2016Interaction Networks for Learning about Objects, Relations and Physicscited by 509
  5. 2021Neural Operator: Learning Maps Between Function Spacescited by 414
  6. 2022PDEBENCH: An Extensive Benchmark for Scientific Machine Learningcited by 413
  7. 2023Geometry-Informed Neural Operator for Large-Scale 3D PDEscited by 303
  8. 2021Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformerscited by 248
  9. 2023Spherical Fourier Neural Operators: Learning Stable Dynamics on the Spherecited by 233
  10. 2023Neural Operators for Accelerating Scientific Simulations and Designcited by 221
  11. 2021StyleSwin: Transformer-based GAN for High-resolution Image Generationcited by 152
  12. 2024DPOT: Auto-Regressive Denoising Operator Transformer for Large-Scale PDE Pre-Trainingcited by 151
  13. 2023Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer Behaviorcited by 131
  14. 2023Scalable Transformer for PDE Surrogate Modelingcited by 128
  15. 2023Multiple Physics Pretraining for Physical Surrogate Modelscited by 98
  16. 2023Multi-Grid Tensorized Fourier Neural Operator for High-Resolution PDEscited by 68
  17. 2022Real-time high-resolution CO2 geological storage prediction using nested Fourier neural operatorscited by 47
  18. 2023Learning Neural PDE Solvers with Parameter-Guided Channel Attentioncited by 38
  19. 2023Self-Supervised Learning with Lie Symmetries for Partial Differential Equationscited by 29

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 12; 2025: 20; 2026: 27.

59 papers.

Sort:

By area:

  1. 2024The Well: a Large-Scale Collection of Diverse Physics Simulations for Machine Learningcited by 145
  2. 2024Continuum Attention for Neural Operatorscited by 40
  3. 2025Walrus: A Cross-Domain Foundation Model for Continuum Dynamicscited by 36
  4. 2024Zero-shot forecasting of chaotic systemscited by 28
  5. 2024FMint: Bridging Human Designed and Data Pretrained Models for Differential Equation Foundation Modelcited by 19
  6. 2025Context parroting: A simple but tough-to-beat baseline for foundation models in scientific machine learningcited by 18
  7. 2024Masked Autoencoders are PDE Learnerscited by 14
  8. 2024Strategies for Pretraining Neural Operatorscited by 14
  9. 2024LeMON: Learning to Learn Multi-Operator Networkscited by 14
  10. 2025Principled Approaches for Extending Neural Architectures to Function Spaces for Operator Learningcited by 12
  11. 2025Lang-PINN: From Language to Physics-Informed Neural Networks via a Multi-Agent Frameworkcited by 11
  12. 2025Operator Learning at Machine Precisioncited by 7
38 more
  1. 2024An Advanced Physics-Informed Neural Operator for Comprehensive Design Optimization of Highly-Nonlinear Systems: An Aerospace Composites Processing Case Studycited by 6
  2. 2025Foundation Models for Scientific Discovery: From Paradigm Enhancement to Paradigm Transitioncited by 6
  3. 2025EddyFormer: Accelerated Neural Simulations of Three-Dimensional Turbulence at Scalecited by 6
  4. 2026CompNO: A Novel Foundation Model approach for solving Partial Differential Equationscited by 5
  5. 2024Data-Efficient Inference of Neural Fluid Fields via SciML Foundation Modelcited by 4
  6. 2025Residuals Are Not Enough: Limits of Physics-Informed Pre-Training for Scientific Foundation Modelscited by 4
  7. 2026Emergent Transfer of a Physics Foundation Model from Simulation to Laboratory Turbulencecited by 4
  8. 2024Liquid Fourier Latent Dynamics Networks for fast GPU-based numerical simulations in computational cardiologycited by 3
  9. 2025Group Downsampling with Equivariant Anti-aliasingcited by 3
  10. 2025F-Adapter: Frequency-Adaptive Parameter-Efficient Fine-Tuning in Scientific Machine Learningcited by 3
  11. 2026Discovering Physical Directions in Weight Space: Composing Neural PDE Expertscited by 3
  12. 2024Learning Lagrangian Interaction Dynamics with Sampling-Based Model Order Reductioncited by 2
  13. 2024M2M: Learning controllable Multi of experts and multi-scale operators are the Partial Differential Equations needcited by 2
  14. 2025Single vs. Multiple Branches in DeepONet and S-DeepONet: Network Architecture Follows Coupling in Multiphysics Systemscited by 2
  15. 2025DRIFT-Net: A Spectral--Coupled Neural Operator for PDEs Learningcited by 2
  16. 2026GenCP: Towards Generative Modeling Paradigm of Coupled Physicscited by 2
  17. 2026Transformers for dynamical systems learn transfer operators in-contextcited by 2
  18. 2026Learning Physical Operators using Neural Operatorscited by 2
  19. 2026Any-Subgroup Equivariant Networks via Symmetry Breakingcited by 2
  20. 2026Chain of Operators: An Inference-Time Harness for In-Context Operator Learningcited by 2
  21. 2025COMPOL: A Unified Neural Operator Framework for Scalable Multi-Physics Simulationscited by 1
  22. 2025SetONet: A Set-Based Operator Network for Solving PDEs with Variable-Input Samplingcited by 1
  23. 2025Maximal Update Parametrization and Zero-Shot Hyperparameter Transfer for Fourier Neural Operatorscited by 1
  24. 2025Scale-Consistent Learning for Partial Differential Equationscited by 1
  25. 2025From Cheap Geometry to Expensive Physics: Elevating Neural Operators via Latent Shape Pretrainingcited by 1
  26. 2026Neuro-Symbolic Multitasking: A Unified Framework for Discovering Generalizable Solutions to PDE Familiescited by 1
  27. 2026Geometry-Aware Post-Hoc Uncertainty Quantification in Operator Learningcited by 1
  28. 2026Extending Fourier Neural Operators for Modeling Parameterized and Coupled PDEscited by 1
  29. 2025SUPRA: Subspace Parameterized Attention for Neural Operator on General Domainscited by 0
  30. 2025Mondrian: Transformer Operators via Domain Decompositioncited by 0
  31. 2025FMint-SDE: A Multimodal Foundation Model for Accelerating Numerical Simulation of SDEs via Error Correctioncited by 0
  32. 2026HGATSolver: A Heterogeneous Graph Attention Solver for Fluid-Structure Interactioncited by 0
  33. 2026Tackling multiphysics problems via finite element-guided physics-informed operator learningcited by 0
  34. 2026Tunable Soft Equivariance with Guaranteescited by 0
  35. 2026Learning Neural Operator Surrogates for the Black Hole Accretion Codecited by 0
  36. 2026An ALE-Consistent Graph Neural Operator-Transformer Framework for Fluid-Structure Interactioncited by 0
  37. 2026MetaColloc: Optimization-Free PDE Solving via Meta-Learned Basis Functionscited by 0
  38. 2026SAFE-SVD: Sensitivity-Aware Fidelity-Enforcing SVD for Physics Foundation Modelscited by 0

50 most cited of 59 All 59 (JSON)