mlsanityData to 7 Oct 2026

OmniArch: Building Foundation Model For Scientific Computing

arXiv 2402.16014 ↗First posted 25 Feb 2024cs.LG cs.AIcited by 8

Not on the map. Why?

Abstract (arXiv)

Foundation models have revolutionized language modeling, while whether this success is replicated in scientific computing remains unexplored. We present OmniArch, the first prototype aiming at solving multi-scale and multi-physics scientific computing problems with physical alignment. We addressed all three challenges with one unified architecture. Its pre-training stage contains a Fourier Encoder-decoder fading out the disharmony across separated dimensions and a Transformer backbone integrating quantities through temporal dynamics, and the novel PDE-Aligner performs physics-informed fine-tuning under flexible conditions. As far as we know, we first conduct 1D-2D-3D united pre-training on the PDEBench, and it sets not only new performance benchmarks for 1D, 2D, and 3D PDEs but also demonstrates exceptional adaptability to new physics via in-context and zero-shot learning approaches, which supports realistic engineering applications and foresight physics discovery.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 30 of its 49 references that are in the map. First posted on arXiv 25 Feb 2024.

Builds on

30 of 49 references.

References · 30

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2017Graph Attention Networkscited by 8,854
  8. 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,806
  9. 2019DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operatorscited by 1,965
  10. 2020Learning Mesh-Based Simulation with Graph Networkscited by 840
  11. 2022FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operatorscited by 748
  12. 2019DeepXDE: A deep learning library for solving differential equationscited by 617
18 more
  1. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  2. 2022Scientific Machine Learning through Physics-Informed Neural Networks: Where we are and What's nextcited by 603
  3. 2022PDEBENCH: An Extensive Benchmark for Scientific Machine Learningcited by 396
  4. 2023Convolutional Neural Operators for robust and accurate learning of PDEscited by 207
  5. 2022Towards Multi-spatiotemporal-scale Generalized PDE Modelingcited by 190
  6. 2023HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolutioncited by 187
  7. 2020U-Net and its variants for medical image segmentation: theory and applicationscited by 169
  8. 2023Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer Behaviorcited by 128
  9. 2023Scalable Transformer for PDE Surrogate Modelingcited by 123
  10. 2023In-Context Operator Learning with Data Prompts for Differential Equation Problemscited by 98
  11. 2023Multiple Physics Pretraining for Physical Surrogate Modelscited by 90
  12. 2023A Practical Survey on Zero-shot Prompt Design for In-context Learningcited by 49
  13. 2023Hypernetwork-based Meta-Learning for Low-Rank Physics-Informed Neural Networkscited by 42
  14. 2022Physical Design using Differentiable Learned Simulatorscited by 37
  15. 2021Meta-Auto-Decoder for Solving Parametric Partial Differential Equationscited by 36
  16. 2023Cross-Modal Fine-Tuning: Align then Refinecited by 34
  17. 2023Self-Supervised Learning with Lie Symmetries for Partial Differential Equationscited by 29
  18. 2024Universal Physics Transformers: A Framework For Efficiently Scaling Neural Operatorscited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 5.

8 papers.

  1. 2024Recent Advances on Machine Learning for Computational Fluid Dynamics: A Surveycited by 27
  2. 2025Physics-informed Temporal Alignment for Auto-regressive PDE Foundation Modelscited by 5
  3. 2025F-Adapter: Frequency-Adaptive Parameter-Efficient Fine-Tuning in Scientific Machine Learningcited by 3
  4. 2026Unsupervised Adaptation of PDE Foundation Modelscited by 1
  5. 2026When Does AI for PDEs Yield Scientific Evidence?cited by 1
  6. 2026UniFluids: Unified Neural Operator Learning with Conditional Flow-matchingcited by 0
  7. 2026Generalized Neural Operator for Parametric and Boundary-Value Problemscited by 0
  8. 2026Does Text Steer Neural PDE Surrogates? A Controlled Diagnostic with OperatorCLIPcited by 0