mlsanityData to 7 Oct 2026

Pretraining a Neural Operator in Lower Dimensions

arXiv 2407.17616 ↗First posted 24 Jul 2024cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

There has recently been increasing attention towards developing foundational neural Partial Differential Equation (PDE) solvers and neural operators through large-scale pretraining. However, unlike vision and language models that make use of abundant and inexpensive (unlabeled) data for pretraining, these neural solvers usually rely on simulated PDE data, which can be costly to obtain, especially for high-dimensional PDEs. In this work, we aim to Pretrain neural PDE solvers on Lower Dimensional PDEs (PreLowD) where data collection is the least expensive. We evaluated the effectiveness of this pretraining strategy in similar PDEs in higher dimensions. We use the Factorized Fourier Neural Operator (FFNO) due to having the necessary flexibility to be applied to PDE data of arbitrary spatial dimensions and reuse trained parameters in lower dimensions. In addition, our work sheds light on the effect of the fine-tuning configuration to make the most of this pretraining strategy. Code is available at https://github.com/BaratiLab/PreLowD.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 42 of its 48 references that are in the map. First posted on arXiv 24 Jul 2024.

Builds on

42 of 48 references.

References · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  4. 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,806
  5. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  6. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  7. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 973
  8. 2019Strategies for Pre-training Graph Neural Networkscited by 964
  9. 2020Neural Operator: Graph Kernel Network for Partial Differential Equationscited by 722
  10. 2020Self-supervised Learning: Generative or Contrastivecited by 693
  11. 2021ImageNet-21K Pretraining for the Massescited by 607
  12. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
30 more
  1. 2020A Survey on Contrastive Self-supervised Learningcited by 553
  2. 2022PDEBENCH: An Extensive Benchmark for Scientific Machine Learningcited by 396
  3. 2018Rethinking ImageNet Pre-trainingcited by 376
  4. 2019Axial Attention in Multidimensional Transformerscited by 350
  5. 2021Choose a Transformer: Fourier or Galerkincited by 333
  6. 2022Transformer for Partial Differential Equations' Operator Learningcited by 270
  7. 2021Factorized Fourier Neural Operatorscited by 237
  8. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  9. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  10. 2021Efficient Visual Pretraining with Contrastive Detectioncited by 150
  11. 2018Self-Supervised Video Representation Learning with Space-Time Cubic Puzzlescited by 149
  12. 2024DPOT: Auto-Regressive Denoising Operator Transformer for Large-Scale PDE Pre-Trainingcited by 137
  13. 2023Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer Behaviorcited by 128
  14. 2023Scalable Transformer for PDE Surrogate Modelingcited by 123
  15. 2023In-Context Operator Learning with Data Prompts for Differential Equation Problemscited by 98
  16. 2023Multiple Physics Pretraining for Physical Surrogate Modelscited by 90
  17. 2020InterBERT: Vision-and-Language Interaction for Multi-modal Pretrainingcited by 48
  18. 2021A Primer on Contrastive Pretraining in Language Processing: Methods, Lessons Learned and Perspectivescited by 41
  19. 2024Data-Efficient Operator Learning via Unsupervised Pretraining and In-Context Learningcited by 32
  20. 2023Self-Supervised Learning with Lie Symmetries for Partial Differential Equationscited by 29
  21. 2024Universal Physics Transformers: A Framework For Efficiently Scaling Neural Operatorscited by 25
  22. 2023Deciphering and integrating invariants for neural operator learning with various physical mechanismscited by 15
  23. 2023Multi-scale Time-stepping of Partial Differential Equations with Transformerscited by 14
  24. 2024Strategies for Pretraining Neural Operatorscited by 14
  25. 2020Efficient Conditional Pre-training for Transfer Learningcited by 11
  26. 2024Masked Autoencoders are PDE Learnerscited by 11
  27. 2023A foundational neural operator that continuously learns without forgettingcited by 10
  28. 2024CaFA: Global Weather Forecasting with Factorized Attention on Spherecited by 6
  29. 2024PICL: Physics Informed Contrastive Learning for Partial Differential Equationscited by 5
  30. 2024Latent Neural PDE Solver: a reduced-order modelling framework for partial differential equationscited by 5

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 3.

5 papers.

  1. 2024Zero-Shot Uncertainty Quantification using Diffusion Probabilistic Modelscited by 14
  2. 2024NeuralDEM -- Real-time Simulation of Industrial Particulate Flowscited by 10
  3. 2025F-Adapter: Frequency-Adaptive Parameter-Efficient Fine-Tuning in Scientific Machine Learningcited by 3
  4. 2025PICore: Physics-Informed Unsupervised Coreset Selection for Data Efficient Neural Operator Trainingcited by 2
  5. 2025Accelerating Data Generation for Nonlinear temporal PDEs via homologous perturbation in solution spacecited by 1