# Hybrid Least Squares/Gradient Descent Methods for DeepONets

arXiv: https://arxiv.org/abs/2508.15394 · HTML: https://mlsanity.com/paper/2508.15394/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 37 references that are in the map.
- First posted on arXiv 21 Aug 2025.

## Builds on (most cited first, 19 of 19)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [Automatic differentiation in machine learning: a survey](https://mlsanity.com/paper/1502.05767/) (2015)
- [Neural Operator: Graph Kernel Network for Partial Differential Equations](https://mlsanity.com/paper/2003.03485/) (2020)
- [Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks](https://mlsanity.com/paper/1602.07868/) (2016)
- [Learning the solution operator of parametric partial differential equations with physics-informed DeepOnets](https://mlsanity.com/paper/2103.10974/) (2021)
- [The Expressive Power of Neural Networks: A View from the Width](https://mlsanity.com/paper/1709.02540/) (2017)
- [Sparse Networks from Scratch: Faster Training without Losing Performance](https://mlsanity.com/paper/1907.04840/) (2019)
- [MIONet: Learning multiple-input operators via tensor product](https://mlsanity.com/paper/2202.06137/) (2022)
- [Universality of Deep Convolutional Neural Networks](https://mlsanity.com/paper/1805.10769/) (2018)
- [ResNet with one-neuron hidden layers is a Universal Approximator](https://mlsanity.com/paper/1806.10909/) (2018)
- [Robust Training and Initialization of Deep Neural Networks: An Adaptive Basis Viewpoint](https://mlsanity.com/paper/1912.04862/) (2019)
- [A Survey on Universal Approximation Theorems](https://mlsanity.com/paper/2407.12895/) (2024)
- [Parareal Neural Networks Emulating a Parallel-in-time Algorithm](https://mlsanity.com/paper/2103.08802/) (2021)
- [Two-level Group Convolution](https://mlsanity.com/paper/2110.05060/) (2021)
- [ELM-DeepONets: Backpropagation-Free Training of Deep Operator Networks via Extreme Learning Machines](https://mlsanity.com/paper/2501.09395/) (2025)
- [Accelerating Deep Learning with Shrinkage and Recall](https://mlsanity.com/paper/1605.01369/) (2016)

## Built on by (most cited first, 2 of 2)

- [The Error of Deep Operator Networks Is the Sum of Its Parts: Branch-Trunk and Mode Error Decompositions](https://mlsanity.com/paper/2602.21910/) (2026)
- [Hybrid Least Squares/Gradient Descent Methods for MIONets](https://mlsanity.com/paper/2607.06976/) (2026)

## Abstract (arXiv)

We propose an efficient hybrid least squares/gradient descent method to accelerate DeepONet training. Since the output of DeepONet can be viewed as linear with respect to the last layer parameters of the branch network, these parameters can be optimized using a least squares (LS) solve, and the remaining hidden layer parameters are updated by means of gradient descent form. However, building the LS system for all possible combinations of branch and trunk inputs yields a prohibitively large linear problem that is infeasible to solve directly. To address this issue, our method decomposes the large LS system into two smaller, more manageable subproblems unicodex2014 one for the branch network and one for the trunk network unicodex2014 and solves them separately. This method is generalized to a broader type of L2 loss with a regularization term for the last layer parameters, including the case of unsupervised learning with physics-informed loss.