mlsanityData to 7 Oct 2026

Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer Behavior

arXiv 2306.00258 ↗First posted 1 Jun 2023cs.LG cs.NA math.NAcited by 128

Not on the map. Why?

Abstract (arXiv)

Pre-trained machine learning (ML) models have shown great performance for a wide range of applications, in particular in natural language processing (NLP) and computer vision (CV). Here, we study how pre-training could be used for scientific machine learning (SciML) applications, specifically in the context of transfer learning. We study the transfer behavior of these models as (i) the pre-trained model size is scaled, (ii) the downstream training dataset size is scaled, (iii) the physics parameters are systematically pushed out of distribution, and (iv) how a single model pre-trained on a mixture of different physics problems can be adapted to various downstream applications. We find that-when fine-tuned appropriately-transfer learning can help reach desired accuracy levels with orders of magnitude fewer downstream examples (across different tasks that can even be out-of-distribution) than training from scratch, with consistent behavior across a wide range of downstream examples. We also find that fine-tuning these models yields more performance gains as model size increases, compared to training from scratch on new downstream tasks. These results hold for a broad range of PDE learning tasks. All in all, our results demonstrate the potential of the "pre-train and fine-tune" paradigm for SciML problems, demonstrating a path towards building SciML foundation models. We open-source our code for reproducibility.

Key facts. Cited by 128 papers in the map (data through 7 Oct 2026). Builds on 37 of its 58 references that are in the map. First posted on arXiv 1 Jun 2023.

Builds on

37 of 58 references.

References · 37

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  7. 2015Domain-Adversarial Training of Neural Networkscited by 3,023
  8. 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,806
  9. 2019DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operatorscited by 1,965
  10. 2014How transferable are features in deep neural networks?cited by 1,216
  11. 2022FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operatorscited by 748
  12. 2017Maximum Classifier Discrepancy for Unsupervised Domain Adaptationcited by 682
25 more
  1. 2021Characterizing possible failure modes in physics-informed neural networkscited by 632
  2. 2021Physics-Informed Neural Operator for Learning Partial Differential Equationscited by 516
  3. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  4. 2022Fourier Neural Operator with Learned Deformations for PDEs on General Geometriescited by 416
  5. 2021Neural Operator: Learning Maps Between Function Spacescited by 413
  6. 2021U-FNO -- An enhanced Fourier neural operator-based deep-learning model for multiphase flowcited by 292
  7. 2020Model Reduction and Neural Networks for Parametric PDEscited by 257
  8. 2022U-NO: U-shaped Neural Operatorscited by 219
  9. 2019Transfer learning enhanced physics informed neural network for phase-field modeling of fracturecited by 116
  10. 2022Learning Operators with Coupled Attentioncited by 88
  11. 2022Deep transfer operator learning for partial differential equations under conditional shiftcited by 83
  12. 2020Transfer learning based multi-fidelity physics informed deep neural networkcited by 68
  13. 2022Reliable extrapolation of deep neural operators informed by physics or sparse observationscited by 62
  14. 2021Seismic wave propagation and inversion with Neural Operatorscited by 51
  15. 2021One-Shot Transfer Learning of Physics-Informed Neural Networkscited by 49
  16. 2022Transfer learning based physics-informed neural networks for solving inverse problems in engineering structures under different loading scenarioscited by 44
  17. 2019Heavy-Tailed Universality Predicts Trends in Test Accuracies for Very Large Pre-Trained Deep Neural Networkscited by 43
  18. 2021Mosaic Flows: A Transferable Deep Learning Framework for Solving PDEs on Unseen Domainscited by 37
  19. 2021Learning elliptic partial differential equations with randomized linear algebracited by 29
  20. 2022Transfer Learning Enhanced DeepONet for Long-Time Prediction of Evolution Equationscited by 28
  21. 2022Explaining the physics of transfer learning a data-driven subgrid-scale closure to a different turbulent flowcited by 20
  22. 2022SVD-PINNs: Transfer Learning of Physics-Informed Neural Networks via Singular Value Decompositioncited by 13
  23. 2020Analysis of three dimensional potential problems in non-homogeneous media with physics-informed deep collocation method using material transfer learning and sensitivity analysiscited by 11
  24. 2021Unsupervised Reservoir Computing for Solving Ordinary Differential Equationscited by 11
  25. 2023Multi-fidelity prediction of fluid flow and temperature field based on transfer learning using Fourier Neural Operatorcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 37; 2025: 44; 2026: 42.

128 papers.

Sort:

By area:

  1. 2023Neural Operators for Accelerating Scientific Simulations and Designcited by 218
  2. 2024Poseidon: Efficient Foundation Models for PDEscited by 186
  3. 2024DPOT: Auto-Regressive Denoising Operator Transformer for Large-Scale PDE Pre-Trainingcited by 137
  4. 2024The Well: a Large-Scale Collection of Diverse Physics Simulations for Machine Learningcited by 134
  5. 2023Multiple Physics Pretraining for Physical Surrogate Modelscited by 90
  6. 2024Pretraining Codomain Attention Neural Operators for Solving Multiphysics PDEscited by 58
  7. 2025CodePDE: An Inference Framework for LLM-driven PDE Solver Generationcited by 37
  8. 2024Advances and Open Challenges in Federated Foundation Modelscited by 33
  9. 2023Artificial General Intelligence for Medical Imaging Analysiscited by 32
  10. 2024Data-Efficient Operator Learning via Unsupervised Pretraining and In-Context Learningcited by 32
  11. 2024Unisolver: PDE-Conditional Transformers Towards Universal Neural PDE Solverscited by 31
  12. 2024PDE Generalization of In-Context Operator Networks: A Study on 1D Scalar Nonlinear Conservation Lawscited by 30
38 more
  1. 2025PDE-Transformer: Efficient and Versatile Transformers for Physics Simulationscited by 29
  2. 2025Generative Latent Neural PDE Solver using Flow Matchingcited by 28
  3. 2024UPS: Efficiently Building Foundation Models for PDE Solving via Cross-Modal Adaptationcited by 27
  4. 2024Recent Advances on Machine Learning for Computational Fluid Dynamics: A Surveycited by 27
  5. 2024Enhancing Foundation Models for Time Series Forecasting via Wavelet-based Tokenizationcited by 26
  6. 2025Walrus: A Cross-Domain Foundation Model for Continuum Dynamicscited by 26
  7. 2024Zero-shot forecasting of chaotic systemscited by 25
  8. 2024Re-Simulation-based Self-Supervised Learning for Pre-Training Foundation Modelscited by 21
  9. 2025PhysiX: A Foundation Model for Physics Simulationscited by 21
  10. 2026Transolver-3: Scaling Up Transformer Solvers to Industrial-Scale Geometriescited by 20
  11. 2024FMint: Bridging Human Designed and Data Pretrained Models for Differential Equation Foundation Modelcited by 18
  12. 2025BCAT: A Block Causal Transformer for PDE Foundation Models for Fluid Dynamicscited by 16
  13. 2025Defining Foundation Models for Computational Science: A Call for Clarity and Rigorcited by 16
  14. 2024PAPERCLIP: Associating Astronomical Observations and Natural Language with Multi-Modal Modelscited by 15
  15. 2024Using Uncertainty Quantification to Characterize and Improve Out-of-Domain Learning for PDEscited by 14
  16. 2024MODNO: Multi Operator Learning With Distributed Neural Operatorscited by 14
  17. 2024Strategies for Pretraining Neural Operatorscited by 14
  18. 2024Zebra: In-Context Generative Pretraining for Solving Parametric PDEscited by 14
  19. 2025Operator Learning: A Statistical Perspectivecited by 14
  20. 2025Context parroting: A simple but tough-to-beat baseline for foundation models in scientific machine learningcited by 14
  21. 2024LeMON: Learning to Learn Multi-Operator Networkscited by 13
  22. 2024Neural Scaling Laws of Deep ReLU and Deep Operator Network: A Theoretical Studycited by 13
  23. 2025Machine learning for modelling unstructured grid data in computational physics: a reviewcited by 12
  24. 2025Panda: A pretrained forecast model for chaotic dynamicscited by 12
  25. 2025MORPH: PDE Foundation Models with Arbitrary Data Modalitycited by 12
  26. 2024Generating synthetic data for neural operatorscited by 11
  27. 2024Masked Autoencoders are PDE Learnerscited by 11
  28. 2024Model Balancing Helps Low-data Training and Fine-tuningcited by 11
  29. 2024DeepONet as a Multi-Operator Extrapolation Model: Distributed Pretraining with Physics-Informed Fine-Tuningcited by 11
  30. 2025Autonomous Agents for Scientific Discovery: Orchestrating Scientists, Language, Code, and Physicscited by 11
  31. 2025Neural equilibria for long-term prediction of nonlinear conservation lawscited by 10
  32. 2025Mixture-of-Experts Operator Transformer for Large-Scale PDE Pre-Trainingcited by 10
  33. 2024Neural Context Flows for Meta-Learning of Dynamical Systemscited by 9
  34. 2024MomentumSMoE: Integrating Momentum into Sparse Mixture of Expertscited by 9
  35. 2025Probabilistic operator learning: generative modeling and uncertainty quantification for foundation models of differential equationscited by 9
  36. 2024OmniArch: Building Foundation Model For Scientific Computingcited by 8
  37. 2025HyPINO: Multi-Physics Neural Operators via HyperPINNs and the Method of Manufactured Solutionscited by 8
  38. 2024GEPS: Boosting Generalization in Parametric PDE Neural Solvers through Adaptive Conditioningcited by 7

50 most cited of 128 All 128 (JSON)