# OmniArch: Building Foundation Model For Scientific Computing

arXiv: https://arxiv.org/abs/2402.16014 · HTML: https://mlsanity.com/paper/2402.16014/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 49 references that are in the map.
- First posted on arXiv 25 Feb 2024.

## Builds on (most cited first, 30 of 30)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [Learning Mesh-Based Simulation with Graph Networks](https://mlsanity.com/paper/2010.03409/) (2020)
- [FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operators](https://mlsanity.com/paper/2202.11214/) (2022)
- [Instruction Tuning for Large Language Models: A Survey](https://mlsanity.com/paper/2308.10792/) (2023)
- [DeepXDE: A deep learning library for solving differential equations](https://mlsanity.com/paper/1907.04502/) (2019)
- [Scientific Machine Learning through Physics-Informed Neural Networks: Where we are and What's next](https://mlsanity.com/paper/2201.05624/) (2022)
- [PDEBENCH: An Extensive Benchmark for Scientific Machine Learning](https://mlsanity.com/paper/2210.07182/) (2022)
- [Convolutional Neural Operators for robust and accurate learning of PDEs](https://mlsanity.com/paper/2302.01178/) (2023)
- [Towards Multi-spatiotemporal-scale Generalized PDE Modeling](https://mlsanity.com/paper/2209.15616/) (2022)
- [HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution](https://mlsanity.com/paper/2306.15794/) (2023)
- [U-Net and its variants for medical image segmentation: theory and applications](https://mlsanity.com/paper/2011.01118/) (2020)
- [Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer Behavior](https://mlsanity.com/paper/2306.00258/) (2023)
- [Scalable Transformer for PDE Surrogate Modeling](https://mlsanity.com/paper/2305.17560/) (2023)
- [In-Context Operator Learning with Data Prompts for Differential Equation Problems](https://mlsanity.com/paper/2304.07993/) (2023)
- [Multiple Physics Pretraining for Physical Surrogate Models](https://mlsanity.com/paper/2310.02994/) (2023)
- [A Practical Survey on Zero-shot Prompt Design for In-context Learning](https://mlsanity.com/paper/2309.13205/) (2023)
- [Hypernetwork-based Meta-Learning for Low-Rank Physics-Informed Neural Networks](https://mlsanity.com/paper/2310.09528/) (2023)
- [Physical Design using Differentiable Learned Simulators](https://mlsanity.com/paper/2202.00728/) (2022)
- [Meta-Auto-Decoder for Solving Parametric Partial Differential Equations](https://mlsanity.com/paper/2111.08823/) (2021)
- [Cross-Modal Fine-Tuning: Align then Refine](https://mlsanity.com/paper/2302.05738/) (2023)
- [Self-Supervised Learning with Lie Symmetries for Partial Differential Equations](https://mlsanity.com/paper/2307.05432/) (2023)
- [Universal Physics Transformers: A Framework For Efficiently Scaling Neural Operators](https://mlsanity.com/paper/2402.12365/) (2024)

## Built on by (most cited first, 9 of 9)

- [Recent Advances on Machine Learning for Computational Fluid Dynamics: A Survey](https://mlsanity.com/paper/2408.12171/) (2024)
- [Physics-informed Temporal Alignment for Auto-regressive PDE Foundation Models](https://mlsanity.com/paper/2505.10930/) (2025)
- [F-Adapter: Frequency-Adaptive Parameter-Efficient Fine-Tuning in Scientific Machine Learning](https://mlsanity.com/paper/2509.23173/) (2025)
- [Generalized Neural Operator for Parametric and Boundary-Value Problems](https://mlsanity.com/paper/2607.21932/) (2026)
- [Unsupervised Adaptation of PDE Foundation Models](https://mlsanity.com/paper/2608.07053/) (2026)
- [When Does AI for PDEs Yield Scientific Evidence?](https://mlsanity.com/paper/2608.22504/) (2026)
- [UniFluids: Unified Neural Operator Learning with Conditional Flow-matching](https://mlsanity.com/paper/2603.22309/) (2026)
- [Does Text Steer Neural PDE Surrogates? A Controlled Diagnostic with OperatorCLIP](https://mlsanity.com/paper/2609.38517/) (2026)
- [SPEAR: A Spectral-Disentangled MoE Neural Operator with Knowledge-Guided Expert Aggregation for Large-Scale PDE Pretraining](https://mlsanity.com/paper/2610.03265/) (2026)

## Abstract (arXiv)

Foundation models have revolutionized language modeling, while whether this success is replicated in scientific computing remains unexplored. We present OmniArch, the first prototype aiming at solving multi-scale and multi-physics scientific computing problems with physical alignment. We addressed all three challenges with one unified architecture. Its pre-training stage contains a Fourier Encoder-decoder fading out the disharmony across separated dimensions and a Transformer backbone integrating quantities through temporal dynamics, and the novel PDE-Aligner performs physics-informed fine-tuning under flexible conditions. As far as we know, we first conduct 1D-2D-3D united pre-training on the PDEBench, and it sets not only new performance benchmarks for 1D, 2D, and 3D PDEs but also demonstrates exceptional adaptability to new physics via in-context and zero-shot learning approaches, which supports realistic engineering applications and foresight physics discovery.