mlsanityData to 7 Oct 2026

Understanding the Expressivity and Trainability of Fourier Neural Operator: A Mean-Field Perspective

arXiv 2310.06379 ↗First posted 10 Oct 2023cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

In this paper, we explores the expressivity and trainability of the Fourier Neural Operator (FNO). We establish a mean-field theory for the FNO, analyzing the behavior of the random FNO from an edge of chaos perspective. Our investigation into the expressivity of a random FNO involves examining the ordered-chaos phase transition of the network based on the weight distribution. This phase transition demonstrates characteristics unique to the FNO, induced by mode truncation, while also showcasing similarities to those of densely connected networks. Furthermore, we identify a connection between expressivity and trainability: the ordered and chaotic phases correspond to regions of vanishing and exploding gradients, respectively. This finding provides a practical prerequisite for the stable training of the FNO. Our experimental results corroborate our theoretical findings.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 29 of its 37 references that are in the map. First posted on arXiv 10 Oct 2023.

Builds on

29 of 37 references.

References · 29

  1. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  2. 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,806
  3. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  4. 2022FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operatorscited by 748
  5. 2020Neural Operator: Graph Kernel Network for Partial Differential Equationscited by 722
  6. 2021Physics-Informed Neural Operator for Learning Partial Differential Equationscited by 516
  7. 2022Fourier Neural Operator with Learned Deformations for PDEs on General Geometriescited by 416
  8. 2022PDEBENCH: An Extensive Benchmark for Scientific Machine Learningcited by 396
  9. 2020Multipole Graph Neural Operator for Parametric Partial Differential Equationscited by 343
  10. 2021U-FNO -- An enhanced Fourier neural operator-based deep-learning model for multiphase flowcited by 292
  11. 2022Transformer for Partial Differential Equations' Operator Learningcited by 270
  12. 2016Exponential expressivity in deep neural networks through transient chaoscited by 260
17 more
  1. 2021Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformerscited by 237
  2. 2021Factorized Fourier Neural Operatorscited by 237
  3. 2022U-NO: U-shaped Neural Operatorscited by 219
  4. 2016Deep Information Propagationcited by 194
  5. 2018Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networkscited by 176
  6. 2017Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practicecited by 132
  7. 2022Clifford Neural Layers for PDE Modelingcited by 91
  8. 2017Mean Field Residual Networks: On the Edge of Chaoscited by 83
  9. 2022Wavelet neural operator: a neural operator for parametric partial differential equationscited by 77
  10. 2022Lie Point Symmetry Data Augmentation for Neural PDE Solverscited by 76
  11. 2018The Emergence of Spectral Universality in Deep Networkscited by 74
  12. 2021Seismic wave propagation and inversion with Neural Operatorscited by 51
  13. 2023Group Equivariant Fourier Neural Operators for Partial Differential Equationscited by 42
  14. 2021Solving PDE-constrained Control Problems Using Operator Learningcited by 41
  15. 2022Transform Once: Efficient Operator Learning in Frequency Domaincited by 20
  16. 2018On the Selection of Initialization and Activation Function for Deep Neural Networkscited by 11
  17. 2019Multiphase flow prediction with deep neural networkscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 4; 2026: 4.

10 papers.

  1. 2025KANO: Kolmogorov-Arnold Neural Operatorcited by 6
  2. 2026Diagnosing Failure Modes of Neural Operators Across Diverse PDE Familiescited by 5
  3. 2024Spectral Convolutional Conditional Neural Processescited by 4
  4. 2024Graph Fourier Neural ODEs: Modeling Spatial-temporal Multi-scales in Molecular Dynamicscited by 2
  5. 2026PDEInvBench: A Comprehensive Dataset and Design Space Exploration of Neural Networks for PDE Inverse Problemscited by 2
  6. 2025Maximal Update Parametrization and Zero-Shot Hyperparameter Transfer for Fourier Neural Operatorscited by 1
  7. 2026Multi-Scale Wavelet Transformers for Operator Learning of Dynamical Systemscited by 1
  8. 2025Analysis of Fourier Neural Operators via Effective Field Theorycited by 0
  9. 2025PGOT: A Physics-Geometry Operator Transformer for Complex PDEscited by 0
  10. 2026A neural operator view on U-Nets for inverse imaging problemscited by 0