# Incremental Spatial and Spectral Learning of Neural Operators for Solving Large-Scale PDEs

arXiv: https://arxiv.org/abs/2211.15188 · HTML: https://mlsanity.com/paper/2211.15188/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 37 references that are in the map.
- First posted on arXiv 28 Nov 2022.

## Builds on (most cited first, 25 of 25)

- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operators](https://mlsanity.com/paper/2202.11214/) (2022)
- [Neural Operator: Graph Kernel Network for Partial Differential Equations](https://mlsanity.com/paper/2003.03485/) (2020)
- [DGM: A deep learning algorithm for solving partial differential equations](https://mlsanity.com/paper/1708.07469/) (2017)
- [Characterizing possible failure modes in physics-informed neural networks](https://mlsanity.com/paper/2109.01050/) (2021)
- [Cyclical Learning Rates for Training Neural Networks](https://mlsanity.com/paper/1506.01186/) (2015)
- [The Deep Ritz method: A deep learning-based numerical algorithm for solving variational problems](https://mlsanity.com/paper/1710.00211/) (2017)
- [Machine learning accelerated computational fluid dynamics](https://mlsanity.com/paper/2102.01010/) (2021)
- [Neural Operator: Learning Maps Between Function Spaces](https://mlsanity.com/paper/2108.08481/) (2021)
- [PDEBENCH: An Extensive Benchmark for Scientific Machine Learning](https://mlsanity.com/paper/2210.07182/) (2022)
- [Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks](https://mlsanity.com/paper/1901.06523/) (2019)
- [U-FNO -- An enhanced Fourier neural operator-based deep-learning model for multiphase flow](https://mlsanity.com/paper/2109.03697/) (2021)
- [Curriculum Learning: A Survey](https://mlsanity.com/paper/2101.10382/) (2021)
- [Bayesian Deep Convolutional Encoder-Decoder Networks for Surrogate Modeling and Uncertainty Quantification](https://mlsanity.com/paper/1801.06879/) (2018)
- [Towards Understanding the Spectral Bias of Deep Learning](https://mlsanity.com/paper/1912.01198/) (2019)
- [Prediction of Aerodynamic Flow Fields Using Convolutional Neural Networks](https://mlsanity.com/paper/1905.13166/) (2019)
- [The Convergence Rate of Neural Networks for Learned Functions of Different Frequencies](https://mlsanity.com/paper/1906.00425/) (2019)
- [The Random Feature Model for Input-Output Maps between Banach Spaces](https://mlsanity.com/paper/2005.10224/) (2020)
- [Learning Operators with Coupled Attention](https://mlsanity.com/paper/2201.01032/) (2022)
- [Learning to Optimize Multigrid PDE Solvers](https://mlsanity.com/paper/1902.10248/) (2019)
- [Splitting Steepest Descent for Growing Neural Architectures](https://mlsanity.com/paper/1910.02366/) (2019)
- [Spectral Bias in Practice: The Role of Function Frequency in Generalization](https://mlsanity.com/paper/2110.02424/) (2021)
- [PINNup: Robust neural network wavefield solutions using frequency upscaling and neuron splitting](https://mlsanity.com/paper/2109.14536/) (2021)

## Built on by (most cited first, 25 of 25)

- [A Library for Learning Neural Operators](https://mlsanity.com/paper/2412.10354/) (2024)
- [Discretization Error of Fourier Neural Operators](https://mlsanity.com/paper/2405.02221/) (2024)
- [PhyMPGN: Physics-encoded Message Passing Graph Network for spatiotemporal PDE systems](https://mlsanity.com/paper/2410.01337/) (2024)
- [Principled Approaches for Extending Neural Architectures to Function Spaces for Operator Learning](https://mlsanity.com/paper/2506.10973/) (2025)
- [InRank: Incremental Low-Rank Learning](https://mlsanity.com/paper/2306.11250/) (2023)
- [Holistic Physics Solver: Learning PDEs in a Unified Spectral-Physical Space](https://mlsanity.com/paper/2410.11382/) (2024)
- [The False Promise of Zero-Shot Super-Resolution in Machine-Learned Operators](https://mlsanity.com/paper/2510.06646/) (2025)
- [A physics-informed transformer neural operator for learning generalized solutions of initial boundary value problems](https://mlsanity.com/paper/2412.09009/) (2024)
- [TensorGRaD: Tensor Gradient Robust Decomposition for Memory-Efficient Neural Operator Training](https://mlsanity.com/paper/2501.02379/) (2025)
- [Enabling Automatic Differentiation with Mollified Graph Neural Operators](https://mlsanity.com/paper/2504.08277/) (2025)
- [Multi-fidelity Fourier Neural Operator for Fast Modeling of Large-Scale Geological Carbon Storage](https://mlsanity.com/paper/2308.09113/) (2023)
- [Bubbleformer: Forecasting Boiling with Transformers](https://mlsanity.com/paper/2507.21244/) (2025)
- [F-Adapter: Frequency-Adaptive Parameter-Efficient Fine-Tuning in Scientific Machine Learning](https://mlsanity.com/paper/2509.23173/) (2025)
- [Method of Manufactured Learning for Solver-free Training of Neural Operators](https://mlsanity.com/paper/2511.12890/) (2025)
- [GeoMaNO: Geometric Mamba Neural Operator for Partial Differential Equations](https://mlsanity.com/paper/2505.12020/) (2025)
- [Learning Physical Operators using Neural Operators](https://mlsanity.com/paper/2602.23113/) (2026)
- [DGNet: Discrete Green Networks for Data-Efficient Learning of Spatiotemporal PDEs](https://mlsanity.com/paper/2603.01762/) (2026)
- [Quantum Circuit Training with Growth-Based Architectures](https://mlsanity.com/paper/2411.16560/) (2024)
- [Maximal Update Parametrization and Zero-Shot Hyperparameter Transfer for Fourier Neural Operators](https://mlsanity.com/paper/2506.19396/) (2025)
- [Accelerating Conjugate Gradient Solvers for Homogenization Problems with Unitary Neural Operators](https://mlsanity.com/paper/2508.02681/) (2025)
- [On the accuracy of implicit neural representations for cardiovascular anatomies and hemodynamic fields](https://mlsanity.com/paper/2510.20970/) (2025)
- [SirenFNO: Efficient and Full Frequency Learning of Fourier Neural Operators](https://mlsanity.com/paper/2606.11518/) (2026)
- [Deep Parallel Spectral Neural Operators for Solving Partial Differential Equations with Enhanced Low-Frequency Learning Capability](https://mlsanity.com/paper/2409.19976/) (2024)
- [Deep Wave Network for Modeling Multi-Scale Physical Dynamics](https://mlsanity.com/paper/2605.04198/) (2026)
- [HiLNO: A Hierarchical Latent Neural Operator with Multi-Scale Supervision for PDEs on General Geometries](https://mlsanity.com/paper/2609.18419/) (2026)

## Abstract (arXiv)

Fourier Neural Operators (FNO) offer a principled approach to solving challenging partial differential equations (PDE) such as turbulent flows. At the core of FNO is a spectral layer that leverages a discretization-convergent representation in the Fourier domain, and learns weights over a fixed set of frequencies. However, training FNO presents two significant challenges, particularly in large-scale, high-resolution applications: (i) Computing Fourier transform on high-resolution inputs is computationally intensive but necessary since fine-scale details are needed for solving many PDEs, such as fluid flows, (ii) selecting the relevant set of frequencies in the spectral layers is challenging, and too many modes can lead to overfitting, while too few can lead to underfitting. To address these issues, we introduce the Incremental Fourier Neural Operator (iFNO), which progressively increases both the number of frequency modes used by the model as well as the resolution of the training data. We empirically show that iFNO reduces total training time while maintaining or improving generalization performance across various datasets. Our method demonstrates a 10% lower testing error, using 20% fewer frequency modes compared to the existing Fourier Neural Operator, while also achieving a 30% faster training.