mlsanityData to 7 Oct 2026

On the training and generalization of deep operator networks

arXiv 2309.01020 ↗First posted 2 Sep 2023math.NA cs.LG cs.NAcited by 31

Not on the map. Why?

Abstract (arXiv)

We present a novel training method for deep operator networks (DeepONets), one of the most popular neural network models for operators. DeepONets are constructed by two sub-networks, namely the branch and trunk networks. Typically, the two sub-networks are trained simultaneously, which amounts to solving a complex optimization problem in a high dimensional space. In addition, the nonconvex and nonlinear nature makes training very challenging. To tackle such a challenge, we propose a two-step training method that trains the trunk network first and then sequentially trains the branch network. The core mechanism is motivated by the divide-and-conquer paradigm and is the decomposition of the entire complex training task into two subtasks with reduced complexity. Therein the Gram-Schmidt orthonormalization process is introduced which significantly improves stability and generalization ability. On the theoretical side, we establish a generalization error estimate in terms of the number of training data, the width of DeepONets, and the number of input and output sensors. Numerical examples are presented to demonstrate the effectiveness of the two-step training method, including Darcy flow in heterogeneous porous media.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 22 of its 46 references that are in the map. First posted on arXiv 2 Sep 2023.

Builds on

22 of 46 references.

References · 22

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  3. 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,806
  4. 2019DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operatorscited by 1,965
  5. 2016An overview of gradient descent optimization algorithmscited by 1,021
  6. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  7. 2020Neural Operator: Graph Kernel Network for Partial Differential Equationscited by 722
  8. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  9. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  10. 2021Factorized Fourier Neural Operatorscited by 237
  11. 2022FourCastNet: Accelerating Global High-Resolution Weather Forecasting using Adaptive Fourier Neural Operatorscited by 188
  12. 2022MIONet: Learning multiple-input operators via tensor productcited by 180
10 more
  1. 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
  2. 2022Learning Operators with Coupled Attentioncited by 88
  3. 2021Improved architectures and training algorithms for deep operator networkscited by 85
  4. 2022Deep transfer operator learning for partial differential equations under conditional shiftcited by 83
  5. 2022Learning two-phase microstructure evolution using neural operators and autoencoder architecturescited by 72
  6. 2021Exponential Convergence of Deep Operator Networks for Elliptic Partial Differential Equationscited by 44
  7. 2021On the Optimal Memorization Power of ReLU Neural Networkscited by 40
  8. 2022SVD Perspectives for Augmenting DeepONet Flexibility and Interpretabilitycited by 35
  9. 2020Provable Memorization via Deep Neural Networks using Sub-linear Parameterscited by 34
  10. 2020Plateau Phenomenon in Gradient Descent Training of ReLU networks: Explanation, Quantification and Avoidancecited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 10; 2025: 12; 2026: 8.

31 papers.

Sort:

By area:

  1. 2024Transformers as Neural Operators for Solutions of Differential Equations with Finite Regularitycited by 33
  2. 2024RiemannONets: Interpretable Neural Operators for Riemann Problemscited by 28
  3. 2023Rethinking materials simulations: Blending direct numerical simulations with neural operatorscited by 25
  4. 2024Basis-to-Basis Operator Learning Using Function Encoderscited by 17
  5. 2024Neural Scaling Laws of Deep ReLU and Deep Operator Network: A Theoretical Studycited by 13
  6. 2024Predicting Crack Nucleation and Propagation in Brittle Materials Using Deep Operator Networks with Diverse Trunk Architecturescited by 12
  7. 2024Ensemble and Mixture-of-Experts DeepONets For Operator Learningcited by 8
  8. 2025Time Marching Neural Operator FE Coupling: AI Accelerated Physics Modelingcited by 8
  9. 2025ATHENA: Agentic Team for Hierarchical Evolutionary Numerical Algorithmscited by 7
  10. 2024Efficient Training of Deep Neural Operator Networks via Randomized Samplingcited by 6
  11. 2025A Variational Framework for Residual-Based Adaptivity in Neural PDE Solvers and Operator Learningcited by 6
  12. 2024Hamilton-Jacobi Based Policy-Iteration via Deep Operator Learningcited by 5
19 more
  1. 2025BubbleOKAN: A Physics-Informed Interpretable Neural Operator for High-Frequency Bubble Dynamicscited by 5
  2. 2026Diagnosing Failure Modes of Neural Operators Across Diverse PDE Familiescited by 5
  3. 2026FEKAN: Feature-Enriched Kolmogorov-Arnold Networkscited by 5
  4. 2025Stochastic Operator Network: A Stochastic Maximum Principle Based Approach to Operator Learningcited by 3
  5. 2025Single vs. Multiple Branches in DeepONet and S-DeepONet: Network Architecture Follows Coupling in Multiphysics Systemscited by 2
  6. 2025AMORE: Adaptive Multi-Output Operator Network for Stiff Chemical Kineticscited by 2
  7. 2025Probabilistic Predictions of Process-Induced Deformation in Carbon/Epoxy Composites Using a Deep Operator Networkcited by 2
  8. 2025Invertible Koopman neural operator for data-driven modeling of partial differential equationscited by 1
  9. 2025Leveraging Deep Operator Networks (DeepONet) for Acoustic Full Waveform Inversion (FWI)cited by 1
  10. 2025Deep operator network for surrogate modeling of poroelasticity with random permeability fieldscited by 1
  11. 2026Physics-Informed Laplace Neural Operator for Solving Partial Differential Equationscited by 1
  12. 2026Spectrally Safe Neural Operator Warm-Starts for Large-Scale Newton Solverscited by 1
  13. 2024A DeepONet for inverting the Neumann-to-Dirichlet Operator in Electrical Impedance Tomography: An approximation theoretic perspective and numerical resultscited by 0
  14. 2024Neural Hamilton: Can A.I. Understand Hamiltonian Mechanics?cited by 0
  15. 2025Crack Path Prediction with Operator Learning using Discrete Particle System data Generationcited by 0
  16. 2026The Error of Deep Operator Networks Is the Sum of Its Parts: Branch-Trunk and Mode Error Decompositionscited by 0
  17. 2026Fourier Neural Operators with Least-Squares Readout Refit for Learning Random Obstacle-to-Solution Mapscited by 0
  18. 2026Two-Step MV-DeepONet: Probabilistic Operator Learning for Uncertainty Propagation Driven by Random Input Fieldscited by 0
  19. 2026Sequential operator learning under dependent datacited by 0