mlsanityData to 7 Oct 2026

Strategies for Pretraining Neural Operators

arXiv 2406.08473 ↗First posted 12 Jun 2024cs.LGcited by 14

Not on the map. Why?

Abstract (arXiv)

Pretraining for partial differential equation (PDE) modeling has recently shown promise in scaling neural operators across datasets to improve generalizability and performance. Despite these advances, our understanding of how pretraining affects neural operators is still limited; studies generally propose tailored architectures and datasets that make it challenging to compare or examine different pretraining frameworks. To address this, we compare various pretraining methods without optimizing architecture choices to characterize pretraining dynamics on different models and datasets as well as to understand its scaling and generalization behavior. We find that pretraining is highly dependent on model and dataset choices, but in general transfer learning or physics-based pretraining strategies work best. In addition, pretraining performance can be further improved by using data augmentations. Lastly, pretraining can be additionally beneficial when fine-tuning in scarce data regimes or when generalizing to downstream data similar to the pretraining distribution. Through providing insights into pretraining neural operators for physics prediction, we hope to motivate future work in developing and evaluating pretraining methods for PDEs.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 52 of its 76 references that are in the map. First posted on arXiv 12 Jun 2024.

Builds on

52 of 76 references.

References · 52

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  9. 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,806
  10. 2019DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operatorscited by 1,965
  11. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  12. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
38 more
  1. 2019A Comprehensive Survey on Transfer Learningcited by 1,154
  2. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  3. 2019Strategies for Pre-training Graph Neural Networkscited by 964
  4. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  5. 2019DeepXDE: A deep learning library for solving differential equationscited by 617
  6. 2021Towards Out-Of-Distribution Generalization: A Surveycited by 431
  7. 2017The Effectiveness of Data Augmentation in Image Classification using Deep Learningcited by 368
  8. 2023Geometry-Informed Neural Operator for Large-Scale 3D PDEscited by 293
  9. 2017Unsupervised Representation Learning by Sorting Sequencescited by 219
  10. 2022Towards Multi-spatiotemporal-scale Generalized PDE Modelingcited by 190
  11. 2024Poseidon: Efficient Foundation Models for PDEscited by 186
  12. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  13. 2018Self-Supervised Video Representation Learning with Space-Time Cubic Puzzlescited by 149
  14. 2024DPOT: Auto-Regressive Denoising Operator Transformer for Large-Scale PDE Pre-Trainingcited by 137
  15. 2023Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer Behaviorcited by 128
  16. 2023Scalable Transformer for PDE Surrogate Modelingcited by 123
  17. 2020Video Playback Rate Perception for Self-supervisedSpatio-Temporal Representation Learningcited by 103
  18. 2023In-Context Operator Learning with Data Prompts for Differential Equation Problemscited by 98
  19. 2019Self-supervised Spatio-temporal Representation Learning for Videos by Predicting Motion and Appearance Statisticscited by 92
  20. 2023Multiple Physics Pretraining for Physical Surrogate Modelscited by 90
  21. 2022Self-Supervised Learning for Videos: A Surveycited by 85
  22. 2022Deep transfer operator learning for partial differential equations under conditional shiftcited by 83
  23. 2022Lie Point Symmetry Data Augmentation for Neural PDE Solverscited by 76
  24. 2024Pretraining Codomain Attention Neural Operators for Solving Multiphysics PDEscited by 58
  25. 2022Transferability in Deep Learning: A Surveycited by 57
  26. 2022Generalizing to New Physical Systems via Context-Informed Dynamics Modelcited by 49
  27. 2023Data-efficient Large Scale Place Recognition with Graded Similarity Supervisioncited by 43
  28. 2021LEADS: Learning Dynamical Systems that Generalize Across Environmentscited by 39
  29. 2022Improving Molecular Contrastive Learning via Faulty Negative Mitigation and Decomposed Fragment Contrastcited by 37
  30. 2024Data-Efficient Operator Learning via Unsupervised Pretraining and In-Context Learningcited by 32
  31. 2023Self-Supervised Learning with Lie Symmetries for Partial Differential Equationscited by 29
  32. 2022MOFormer: Self-Supervised Transformer model for Metal-Organic Framework Property Predictioncited by 28
  33. 2023Physics Informed Token Transformer for Solving Partial Differential Equationscited by 21
  34. 2023PROSE: Predicting Operators and Symbolic Expressions using Multimodal Transformerscited by 16
  35. 2023Deciphering and integrating invariants for neural operator learning with various physical mechanismscited by 15
  36. 2024Masked Autoencoders are PDE Learnerscited by 11
  37. 2023A foundational neural operator that continuously learns without forgettingcited by 10
  38. 2024PICL: Physics Informed Contrastive Learning for Partial Differential Equationscited by 5

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 7; 2026: 2.

14 papers.

Sort:

By area:

  1. 2024Text2PDE: Latent Diffusion Models for Accessible Physics Simulationcited by 25
  2. 2024Zero-Shot Uncertainty Quantification using Diffusion Probabilistic Modelscited by 14
  3. 2024Explain Like I'm Five: Using LLMs to Improve PDE Surrogate Models with Textcited by 6
  4. 2024Pretraining a Neural Operator in Lower Dimensionscited by 5
  5. 2025Physics-informed Temporal Alignment for Auto-regressive PDE Foundation Modelscited by 5
  6. 2024Model-Agnostic AI Framework with Explicit Time Integration for Long-Term Fluid Dynamics Predictioncited by 3
  7. 2025F-Adapter: Frequency-Adaptive Parameter-Efficient Fine-Tuning in Scientific Machine Learningcited by 3
  8. 2025Method of Manufactured Learning for Solver-free Training of Neural Operatorscited by 3
  9. 2025Reframing Generative Models for Physical Systems using Stochastic Interpolantscited by 2
  10. 2025Convolutional-neural-operator-based transfer learning for solving PDEscited by 2
  11. 2025ATOM: A Pretrained Neural Operator for Multitask Molecular Dynamicscited by 1
  12. 2025Accelerating Data Generation for Nonlinear temporal PDEs via homologous perturbation in solution spacecited by 1
2 more
  1. 2026MeshTok: Efficient Multi-Scale Tokenization for Scalable PDE Transformerscited by 0
  2. 2026Cross-Domain Pretraining for Steady-State Neural CFD Surrogatescited by 0