mlsanityData to 7 Oct 2026

DPOT: Auto-Regressive Denoising Operator Transformer for Large-Scale PDE Pre-Training

arXiv 2403.03542 ↗First posted 6 Mar 2024cs.LG cs.NA math.NAcited by 137

Not on the map. Why?

Abstract (arXiv)

Pre-training has been investigated to improve the efficiency and performance of training neural operators in data-scarce settings. However, it is largely in its infancy due to the inherent complexity and diversity, such as long trajectories, multiple scales and varying dimensions of partial differential equations (PDEs) data. In this paper, we present a new auto-regressive denoising pre-training strategy, which allows for more stable and efficient pre-training on PDE data and generalizes to various downstream tasks. Moreover, by designing a flexible and scalable model architecture based on Fourier attention, we can easily scale up the model for large-scale pre-training. We train our PDE foundation model with up to 0.5B parameters on 10+ PDE datasets with more than 100k trajectories. Extensive experiments show that we achieve SOTA on these benchmarks and validate the strong generalizability of our model to significantly enhance performance on diverse downstream PDE tasks like 3D data. Code is available at urlhttps://github.com/thu-ml/DPOT.

Key facts. Cited by 137 papers in the map (data through 7 Oct 2026). Builds on 38 of its 55 references that are in the map. First posted on arXiv 6 Mar 2024.

Builds on

38 of 55 references.

References · 38

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  6. 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,806
  7. 2019DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operatorscited by 1,965
  8. 2018Group Normalizationcited by 1,514
  9. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  10. 2022FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operatorscited by 748
  11. 2022Scientific Machine Learning through Physics-Informed Neural Networks: Where we are and What's nextcited by 603
  12. 2021Learning the solution operator of parametric partial differential equations with physics-informed DeepOnetscited by 537
26 more
  1. 2021Physics-Informed Neural Operator for Learning Partial Differential Equationscited by 516
  2. 2022PDEBENCH: An Extensive Benchmark for Scientific Machine Learningcited by 396
  3. 2022Message Passing Neural PDE Solverscited by 368
  4. 2021Choose a Transformer: Fourier or Galerkincited by 333
  5. 2023ClimaX: A foundation model for weather and climatecited by 325
  6. 2023GNOT: A General Neural Operator Transformer for Operator Learningcited by 321
  7. 2023Geometry-Informed Neural Operator for Large-Scale 3D PDEscited by 293
  8. 2022Transformer for Partial Differential Equations' Operator Learningcited by 270
  9. 2021Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformerscited by 237
  10. 2021Factorized Fourier Neural Operatorscited by 237
  11. 2023Neural Operators for Accelerating Scientific Simulations and Designcited by 218
  12. 2022Towards Multi-spatiotemporal-scale Generalized PDE Modelingcited by 190
  13. 2019Variational Physics-Informed Neural Networks For Solving Partial Differential Equationscited by 144
  14. 2023Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer Behaviorcited by 128
  15. 2019Bridging the Gap between Training and Inference for Neural Machine Translationcited by 123
  16. 2023In-Context Operator Learning with Data Prompts for Differential Equation Problemscited by 98
  17. 2023Multiple Physics Pretraining for Physical Surrogate Modelscited by 90
  18. 2023Solving High-Dimensional PDEs with Latent Spectral Modelscited by 84
  19. 2021The Modern Mathematics of Deep Learningcited by 63
  20. 2023Neural Operator Learning for Long-Time Integration in Dynamical Systems with Recurrent Neural Networkscited by 32
  21. 2023Sumformer: Universal Approximation for Efficient Transformerscited by 30
  22. 2023Self-Supervised Learning with Lie Symmetries for Partial Differential Equationscited by 29
  23. 2023AI Foundation Models for Weather and Climate: Applications, Design, and Implementationcited by 19
  24. 2023A neural operator-based surrogate solver for free-form electromagnetic inverse designcited by 18
  25. 2023RecFNO: a resolution-invariant flow and heat field reconstruction method from sparse observations via Fourier neural operatorcited by 17
  26. 2023NUNO: A General Framework for Learning Parametric PDEs with Non-Uniform Datacited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 22; 2025: 48; 2026: 67.

137 papers.

Sort:

By area:

  1. 2024Poseidon: Efficient Foundation Models for PDEscited by 186
  2. 2024Pretraining Codomain Attention Neural Operators for Solving Multiphysics PDEscited by 58
  3. 2024CViT: Continuous Vision Transformer for Operator Learningcited by 47
  4. 2025CodePDE: An Inference Framework for LLM-driven PDE Solver Generationcited by 37
  5. 2024Unisolver: PDE-Conditional Transformers Towards Universal Neural PDE Solverscited by 31
  6. 2024On the Benefits of Memory for Modeling Time-Dependent PDEscited by 30
  7. 2024UPS: Efficiently Building Foundation Models for PDE Solving via Cross-Modal Adaptationcited by 27
  8. 2024Recent Advances on Machine Learning for Computational Fluid Dynamics: A Surveycited by 27
  9. 2025Walrus: A Cross-Domain Foundation Model for Continuum Dynamicscited by 26
  10. 2024Text2PDE: Latent Diffusion Models for Accessible Physics Simulationcited by 25
  11. 2025DISCO: learning to DISCover an evolution Operator for multi-physics-agnostic predictioncited by 21
  12. 2026Transolver-3: Scaling Up Transformer Solvers to Industrial-Scale Geometriescited by 20
38 more
  1. 2025FunDiff: Diffusion Models over Function Spaces for Physics-Informed Generative Modelingcited by 19
  2. 2025BCAT: A Block Causal Transformer for PDE Foundation Models for Fluid Dynamicscited by 16
  3. 2024Strategies for Pretraining Neural Operatorscited by 14
  4. 2024Zebra: In-Context Generative Pretraining for Solving Parametric PDEscited by 14
  5. 2026RealPDEBench: A Benchmark for Complex Physical Systems with Real-World Datacited by 13
  6. 2025MORPH: PDE Foundation Models with Arbitrary Data Modalitycited by 12
  7. 2024Uncertainty Quantification of Surrogate Models using Conformal Predictioncited by 11
  8. 2024Model Balancing Helps Low-data Training and Fine-tuningcited by 11
  9. 2025Principled Approaches for Extending Neural Architectures to Function Spaces for Operator Learningcited by 11
  10. 2024Spectral-Refiner: Accurate Fine-Tuning of Spatiotemporal Fourier Neural Operator for Turbulent Flowscited by 10
  11. 2024Accelerating Phase Field Simulations Through a Hybrid Adaptive Fourier Neural Operator with U-Net Backbonecited by 10
  12. 2024Holistic Physics Solver: Learning PDEs in a Unified Spectral-Physical Spacecited by 10
  13. 2025Mixture-of-Experts Operator Transformer for Large-Scale PDE Pre-Trainingcited by 10
  14. 2025Eigenspectrum Analysis of Neural Networks without Aspect Ratio Biascited by 9
  15. 2025Probabilistic operator learning: generative modeling and uncertainty quantification for foundation models of differential equationscited by 9
  16. 2024Predicting Change, Not States: An Alternate Framework for Neural PDE Surrogatescited by 8
  17. 2025Latent Mamba Operator for Partial Differential Equationscited by 8
  18. 2025FourierFlow: Frequency-aware Flow Matching for Generative Turbulence Modelingcited by 8
  19. 2025PhysicsCorrect: A Training-Free Approach for Stable Neural PDE Simulationscited by 8
  20. 2025HyPINO: Multi-Physics Neural Operators via HyperPINNs and the Method of Manufactured Solutionscited by 8
  21. 2026Test-time Generalization for Physics through Neural Operator Splittingcited by 8
  22. 2026GeoPT: Scaling Physics Simulation via Lifted Geometric Pre-Trainingcited by 8
  23. 2025TANTE: Time-Adaptive Operator Learning via Neural Taylor Expansioncited by 7
  24. 2025Light-Weight Diffusion Multiplier and Uncertainty Quantification for Fourier Neural Operatorscited by 6
  25. 2025SAOT: An Enhanced Locality-Aware Spectral Transformer for Solving PDEscited by 6
  26. 2025PI-MFM: Physics-informed multimodal foundation model for solving partial differential equationscited by 6
  27. 2026Learning PDE Solvers with Physics and Data: A Unifying View of Physics-Informed Neural Networks and Neural Operatorscited by 6
  28. 2024Pretraining a Neural Operator in Lower Dimensionscited by 5
  29. 2024Micrometer: Micromechanics Transformer for Predicting Mechanical Responses of Heterogeneous Materialscited by 5
  30. 2024Latent Neural Operator Pretraining for Solving Time-Dependent PDEscited by 5
  31. 2025Physics-informed Temporal Alignment for Auto-regressive PDE Foundation Modelscited by 5
  32. 2025Towards a Physics Foundation Modelcited by 5
  33. 2026Probabilistic Retrofitting of Learned Simulatorscited by 5
  34. 2024Data-Efficient Inference of Neural Fluid Fields via SciML Foundation Modelcited by 4
  35. 2024MATEY: multiscale adaptive foundation models for spatiotemporal physical systemscited by 4
  36. 2025FD-Bench: A Modular and Fair Benchmark for Data-driven Fluid Simulationcited by 4
  37. 2026OpInf-LLM: Parametric PDE Solving with LLMs via Operator Inferencecited by 4
  38. 2026FlowRefiner: Flow Matching-Based Iterative Refinement for 3D Turbulent Flow Simulationcited by 4

50 most cited of 137 All 137 (JSON)