mlsanityData to 7 Oct 2026

NESTOR: A Nested MOE-based Neural Operator for Large-Scale PDE Pre-Training

arXiv 2602.22059 ↗First posted 25 Feb 2026cs.CV cs.AIcited by 1

On the map: Mixture-of-experts → MoE theory and gating functions

Abstract (arXiv)

Neural operators have emerged as an efficient paradigm for solving PDEs, overcoming the limitations of traditional numerical methods and significantly improving computational efficiency. However, due to the diversity and complexity of PDE systems, existing neural operators typically rely on a single network architecture, which limits their capacity to fully capture heterogeneous features and complex system dependencies. This constraint poses a bottleneck for large-scale PDE pre-training based on neural operators. To address these challenges, we propose a large-scale PDE pre-trained neural operator based on a nested Mixture-of-Experts (MoE) framework. In particular, the image-level MoE is designed to capture global dependencies, while the token-level Sub-MoE focuses on local dependencies. Our model can selectively activate the most suitable expert networks for a given input, thereby enhancing generalization and transferability. We conduct large-scale pre-training on twelve PDE datasets from diverse sources and successfully transfer the model to downstream tasks. Extensive experiments demonstrate the effectiveness of our approach.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts. Builds on 24 of its 34 references that are in the map. First posted on arXiv 25 Feb 2026.

Builds on

24 of 34 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Mixture-of-Experts Operator Transformer for Large-Scale PDE Pre-Training29 Oct

Other references · 23

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  7. 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,806
  8. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  9. 2022FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operatorscited by 748
  10. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  11. 2021Neural Operator: Learning Maps Between Function Spacescited by 413
  12. 2022PDEBENCH: An Extensive Benchmark for Scientific Machine Learningcited by 396
11 more
  1. 2021Choose a Transformer: Fourier or Galerkincited by 333
  2. 2023GNOT: A General Neural Operator Transformer for Operator Learningcited by 321
  3. 2021Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformerscited by 237
  4. 2022Towards Multi-spatiotemporal-scale Generalized PDE Modelingcited by 190
  5. 2024Poseidon: Efficient Foundation Models for PDEscited by 186
  6. 2024DPOT: Auto-Regressive Denoising Operator Transformer for Large-Scale PDE Pre-Trainingcited by 137
  7. 2023Multiple Physics Pretraining for Physical Surrogate Modelscited by 90
  8. 2023CFDBench: A Large-Scale Benchmark for Machine Learning Methods in Fluid Dynamicscited by 40
  9. 2024Unisolver: PDE-Conditional Transformers Towards Universal Neural PDE Solverscited by 31
  10. 2023ViTO: Vision Transformer-Operatorcited by 29
  11. 2024HMoE: Heterogeneous Mixture of Experts for Language Modelingcited by 28

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Temperature Field Reconstruction of Tungsten Monoblock Divertor on EAST using Physics-aware Neural Operator Transformercited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026SPEAR: A Spectral-Disentangled MoE Neural Operator with Knowledge-Guided Expert Aggregation for Large-Scale PDE Pretraining
  2. 2026COSMOS: Soft Mechanism Mixtures with Verifiable Routing for Long-Horizon PDE Forecasting
  3. 2026Latent-MoE: Domain-Aware Mixture-of-Experts for PDEs with Multi-Regime Physics
  4. 2025Mixture of neural operator experts for learning boundary conditions and model selection
  5. 2026UniFluids: Unified Neural Operator Learning with Conditional Flow-matching
  6. 2025Symphony-MoE: Harmonizing Disparate Pre-trained Models into a Coherent Mixture-of-Experts
  7. 2026OpInf-LLM: Parametric PDE Solving with LLMs via Operator Inference
  8. 2026Localized Operator Learning with Adaptive Partition-of-Unity Mixture-of-Expert Networks