# On the training and generalization of deep operator networks

arXiv: https://arxiv.org/abs/2309.01020 · HTML: https://mlsanity.com/paper/2309.01020/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 46 references that are in the map.
- First posted on arXiv 2 Sep 2023.

## Builds on (most cited first, 22 of 22)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [A Convergence Theory for Deep Learning via Over-Parameterization](https://mlsanity.com/paper/1811.03962/) (2018)
- [Neural Operator: Graph Kernel Network for Partial Differential Equations](https://mlsanity.com/paper/2003.03485/) (2020)
- [Gradient Descent Provably Optimizes Over-parameterized Neural Networks](https://mlsanity.com/paper/1810.02054/) (2018)
- [Gradient Descent Finds Global Minima of Deep Neural Networks](https://mlsanity.com/paper/1811.03804/) (2018)
- [Factorized Fourier Neural Operators](https://mlsanity.com/paper/2111.13802/) (2021)
- [FourCastNet: Accelerating Global High-Resolution Weather Forecasting using Adaptive Fourier Neural Operators](https://mlsanity.com/paper/2208.05419/) (2022)
- [MIONet: Learning multiple-input operators via tensor product](https://mlsanity.com/paper/2202.06137/) (2022)
- [Towards moderate overparameterization: global convergence guarantees for training shallow neural networks](https://mlsanity.com/paper/1902.04674/) (2019)
- [Learning Operators with Coupled Attention](https://mlsanity.com/paper/2201.01032/) (2022)
- [Improved architectures and training algorithms for deep operator networks](https://mlsanity.com/paper/2110.01654/) (2021)
- [Deep transfer operator learning for partial differential equations under conditional shift](https://mlsanity.com/paper/2204.09810/) (2022)
- [Learning two-phase microstructure evolution using neural operators and autoencoder architectures](https://mlsanity.com/paper/2204.07230/) (2022)
- [Exponential Convergence of Deep Operator Networks for Elliptic Partial Differential Equations](https://mlsanity.com/paper/2112.08125/) (2021)
- [On the Optimal Memorization Power of ReLU Neural Networks](https://mlsanity.com/paper/2110.03187/) (2021)
- [SVD Perspectives for Augmenting DeepONet Flexibility and Interpretability](https://mlsanity.com/paper/2204.12670/) (2022)
- [Provable Memorization via Deep Neural Networks using Sub-linear Parameters](https://mlsanity.com/paper/2010.13363/) (2020)
- [Plateau Phenomenon in Gradient Descent Training of ReLU networks: Explanation, Quantification and Avoidance](https://mlsanity.com/paper/2007.07213/) (2020)

## Built on by (most cited first, 30 of 31)

- [Transformers as Neural Operators for Solutions of Differential Equations with Finite Regularity](https://mlsanity.com/paper/2405.19166/) (2024)
- [RiemannONets: Interpretable Neural Operators for Riemann Problems](https://mlsanity.com/paper/2401.08886/) (2024)
- [Rethinking materials simulations: Blending direct numerical simulations with neural operators](https://mlsanity.com/paper/2312.05410/) (2023)
- [Basis-to-Basis Operator Learning Using Function Encoders](https://mlsanity.com/paper/2410.00171/) (2024)
- [Neural Scaling Laws of Deep ReLU and Deep Operator Network: A Theoretical Study](https://mlsanity.com/paper/2410.00357/) (2024)
- [Predicting Crack Nucleation and Propagation in Brittle Materials Using Deep Operator Networks with Diverse Trunk Architectures](https://mlsanity.com/paper/2501.00016/) (2024)
- [Ensemble and Mixture-of-Experts DeepONets For Operator Learning](https://mlsanity.com/paper/2405.11907/) (2024)
- [Time Marching Neural Operator FE Coupling: AI Accelerated Physics Modeling](https://mlsanity.com/paper/2504.11383/) (2025)
- [ATHENA: Agentic Team for Hierarchical Evolutionary Numerical Algorithms](https://mlsanity.com/paper/2512.03476/) (2025)
- [Efficient Training of Deep Neural Operator Networks via Randomized Sampling](https://mlsanity.com/paper/2409.13280/) (2024)
- [A Variational Framework for Residual-Based Adaptivity in Neural PDE Solvers and Operator Learning](https://mlsanity.com/paper/2509.14198/) (2025)
- [Diagnosing Failure Modes of Neural Operators Across Diverse PDE Families](https://mlsanity.com/paper/2601.11428/) (2026)
- [Hamilton-Jacobi Based Policy-Iteration via Deep Operator Learning](https://mlsanity.com/paper/2406.10920/) (2024)
- [BubbleOKAN: A Physics-Informed Interpretable Neural Operator for High-Frequency Bubble Dynamics](https://mlsanity.com/paper/2508.03965/) (2025)
- [FEKAN: Feature-Enriched Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2602.16530/) (2026)
- [Stochastic Operator Network: A Stochastic Maximum Principle Based Approach to Operator Learning](https://mlsanity.com/paper/2507.10401/) (2025)
- [Probabilistic Predictions of Process-Induced Deformation in Carbon/Epoxy Composites Using a Deep Operator Network](https://mlsanity.com/paper/2512.13746/) (2025)
- [Single vs. Multiple Branches in DeepONet and S-DeepONet: Network Architecture Follows Coupling in Multiphysics Systems](https://mlsanity.com/paper/2507.03660/) (2025)
- [AMORE: Adaptive Multi-Output Operator Network for Stiff Chemical Kinetics](https://mlsanity.com/paper/2510.12999/) (2025)
- [Invertible Koopman neural operator for data-driven modeling of partial differential equations](https://mlsanity.com/paper/2503.19717/) (2025)
- [Leveraging Deep Operator Networks (DeepONet) for Acoustic Full Waveform Inversion (FWI)](https://mlsanity.com/paper/2504.10720/) (2025)
- [Deep operator network for surrogate modeling of poroelasticity with random permeability fields](https://mlsanity.com/paper/2509.11966/) (2025)
- [Physics-Informed Laplace Neural Operator for Solving Partial Differential Equations](https://mlsanity.com/paper/2602.12706/) (2026)
- [Spectrally Safe Neural Operator Warm-Starts for Large-Scale Newton Solvers](https://mlsanity.com/paper/2606.21828/) (2026)
- [A DeepONet for inverting the Neumann-to-Dirichlet Operator in Electrical Impedance Tomography: An approximation theoretic perspective and numerical results](https://mlsanity.com/paper/2407.17182/) (2024)
- [Neural Hamilton: Can A.I. Understand Hamiltonian Mechanics?](https://mlsanity.com/paper/2410.20951/) (2024)
- [Crack Path Prediction with Operator Learning using Discrete Particle System data Generation](https://mlsanity.com/paper/2506.01976/) (2025)
- [The Error of Deep Operator Networks Is the Sum of Its Parts: Branch-Trunk and Mode Error Decompositions](https://mlsanity.com/paper/2602.21910/) (2026)
- [Fourier Neural Operators with Least-Squares Readout Refit for Learning Random Obstacle-to-Solution Maps](https://mlsanity.com/paper/2606.29436/) (2026)
- [Two-Step MV-DeepONet: Probabilistic Operator Learning for Uncertainty Propagation Driven by Random Input Fields](https://mlsanity.com/paper/2608.09071/) (2026)

## Abstract (arXiv)

We present a novel training method for deep operator networks (DeepONets), one of the most popular neural network models for operators. DeepONets are constructed by two sub-networks, namely the branch and trunk networks. Typically, the two sub-networks are trained simultaneously, which amounts to solving a complex optimization problem in a high dimensional space. In addition, the nonconvex and nonlinear nature makes training very challenging. To tackle such a challenge, we propose a two-step training method that trains the trunk network first and then sequentially trains the branch network. The core mechanism is motivated by the divide-and-conquer paradigm and is the decomposition of the entire complex training task into two subtasks with reduced complexity. Therein the Gram-Schmidt orthonormalization process is introduced which significantly improves stability and generalization ability. On the theoretical side, we establish a generalization error estimate in terms of the number of training data, the width of DeepONets, and the number of input and output sensors. Numerical examples are presented to demonstrate the effectiveness of the two-step training method, including Darcy flow in heterogeneous porous media.