# On the influence of over-parameterization in manifold based surrogates and deep neural operators

arXiv: https://arxiv.org/abs/2203.05071 · HTML: https://mlsanity.com/paper/2203.05071/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 61 references that are in the map.
- First posted on arXiv 9 Mar 2022.

## Builds on (most cited first, 18 of 18)

- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [Self-Adaptive Physics-Informed Neural Networks using a Soft Attention Mechanism](https://mlsanity.com/paper/2009.04544/) (2020)
- [The Computational Limits of Deep Learning](https://mlsanity.com/paper/2007.05558/) (2020)
- [A Universal Law of Robustness via Isoperimetry](https://mlsanity.com/paper/2105.12806/) (2021)
- [On the Power of Over-parametrization in Neural Networks with Quadratic Activation](https://mlsanity.com/paper/1803.01206/) (2018)
- [A physics-informed variational DeepONet for predicting the crack path in brittle materials](https://mlsanity.com/paper/2108.06905/) (2021)
- [Numerical Gaussian Processes for Time-dependent and Non-linear Partial Differential Equations](https://mlsanity.com/paper/1703.10230/) (2017)
- [Theoretical Issues in Deep Networks: Approximation, Optimization and Generalization](https://mlsanity.com/paper/1908.09375/) (2019)
- [A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learning](https://mlsanity.com/paper/2109.02355/) (2021)
- [Deep neural networks for the evaluation and design of photonic devices](https://mlsanity.com/paper/2007.00084/) (2020)
- [A survey of unsupervised learning methods for high-dimensional uncertainty quantification in black-box-type problems](https://mlsanity.com/paper/2202.04648/) (2022)
- [Extending classical surrogate modelling to high-dimensions through supervised dimensionality reduction: a data-driven approach](https://mlsanity.com/paper/1812.06309/) (2018)
- [Manifold learning-based polynomial chaos expansions for high-dimensional surrogate models](https://mlsanity.com/paper/2107.09814/) (2021)
- [Sparse Polynomial Chaos Expansions via Compressed Sensing and D-optimal Design](https://mlsanity.com/paper/1712.10131/) (2017)
- [Compressive sensing adaptation for polynomial chaos expansions](https://mlsanity.com/paper/1801.01961/) (2018)
- [Robust Adaptive Least Squares Polynomial Chaos Expansions in High-Frequency Applications](https://mlsanity.com/paper/1912.07725/) (2019)

## Built on by (most cited first, 29 of 29)

- [Physics-Informed Deep Neural Operator Networks](https://mlsanity.com/paper/2207.05748/) (2022)
- [Physics-Informed Machine Learning: A Survey on Problems, Methods and Applications](https://mlsanity.com/paper/2211.08064/) (2022)
- [Deep transfer operator learning for partial differential equations under conditional shift](https://mlsanity.com/paper/2204.09810/) (2022)
- [SVD Perspectives for Augmenting DeepONet Flexibility and Interpretability](https://mlsanity.com/paper/2204.12670/) (2022)
- [Learning stiff chemical kinetics using extended deep neural operators](https://mlsanity.com/paper/2302.12645/) (2023)
- [Neural Chaos: A Spectral Stochastic Neural Operator](https://mlsanity.com/paper/2502.11835/) (2025)
- [Deep neural operators can serve as accurate surrogates for shape optimization: A case study for airfoils](https://mlsanity.com/paper/2302.00807/) (2023)
- [Learning in latent spaces improves the predictive accuracy of deep neural operators](https://mlsanity.com/paper/2304.07599/) (2023)
- [Synergistic Learning with Multi-Task DeepONet for Efficient PDE Problem Solving](https://mlsanity.com/paper/2408.02198/) (2024)
- [Basis-to-Basis Operator Learning Using Function Encoders](https://mlsanity.com/paper/2410.00171/) (2024)
- [Separable DeepONet: Breaking the Curse of Dimensionality in Physics-Informed Machine Learning](https://mlsanity.com/paper/2407.15887/) (2024)
- [Sound propagation in realistic interactive 3D scenes with parameterized sources using deep neural operators](https://mlsanity.com/paper/2308.05141/) (2023)
- [A Hybrid Deep Neural Operator/Finite Element Method for Ice-Sheet Modeling](https://mlsanity.com/paper/2301.11402/) (2023)
- [Physics-aware Machine Learning Revolutionizes Scientific Paradigm for Machine Learning and Process-based Hydrology](https://mlsanity.com/paper/2310.05227/) (2023)
- [Physics-Informed Latent Neural Operator for Real-time Predictions of time-dependent parametric PDEs](https://mlsanity.com/paper/2501.08428/) (2025)
- [Neural Operator: Is data all you need to model the world? An insight into the paradigm of data-driven scientific ML](https://mlsanity.com/paper/2301.13331/) (2023)
- [Polynomial Chaos Expansion for Operator Learning](https://mlsanity.com/paper/2508.20886/) (2025)
- [Polynomial Chaos Expansions on Principal Geodesic Grassmannian Submanifolds for Surrogate Modeling and Uncertainty Quantification](https://mlsanity.com/paper/2401.16683/) (2024)
- [Efficient Training of Deep Neural Operator Networks via Randomized Sampling](https://mlsanity.com/paper/2409.13280/) (2024)
- [Neural Operators for Stochastic Modeling of Nonlinear Structural System Response to Natural Hazards](https://mlsanity.com/paper/2502.11279/) (2025)
- [A Framework for Strategic Discovery of Credible Neural Network Surrogate Models under Uncertainty](https://mlsanity.com/paper/2403.08901/) (2024)
- [Physics-constrained 3D Convolutional Neural Networks for Electrodynamics](https://mlsanity.com/paper/2301.13715/) (2023)
- [Alternators For Sequence Modeling](https://mlsanity.com/paper/2405.11848/) (2024)
- [Dimensionality reduction can be used as a surrogate model for high-dimensional forward uncertainty quantification](https://mlsanity.com/paper/2402.04582/) (2024)
- [AMORE: Adaptive Multi-Output Operator Network for Stiff Chemical Kinetics](https://mlsanity.com/paper/2510.12999/) (2025)
- [DON-LSTM: Multi-Resolution Learning with DeepONets and Long Short-Term Memory Neural Networks](https://mlsanity.com/paper/2310.02491/) (2023)
- [Physics-Informed Laplace Neural Operator for Solving Partial Differential Equations](https://mlsanity.com/paper/2602.12706/) (2026)
- [A novel Fourier neural operator framework for classification of multi-sized images: Application to three dimensional digital porous media](https://mlsanity.com/paper/2402.11568/) (2024)
- [A Resolution Independent Neural Operator](https://mlsanity.com/paper/2407.13010/) (2024)

## Abstract (arXiv)

Constructing accurate and generalizable approximators for complex physico-chemical processes exhibiting highly non-smooth dynamics is challenging. In this work, we propose new developments and perform comparisons for two promising approaches: manifold-based polynomial chaos expansion (m-PCE) and the deep neural operator (DeepONet), and we examine the effect of over-parameterization on generalization. We demonstrate the performance of these methods in terms of generalization accuracy by solving the 2D time-dependent Brusselator reaction-diffusion system with uncertainty sources, modeling an autocatalytic chemical reaction between two species. We first propose an extension of the m-PCE by constructing a mapping between latent spaces formed by two separate embeddings of input functions and output QoIs. To enhance the accuracy of the DeepONet, we introduce weight self-adaptivity in the loss function. We demonstrate that the performance of m-PCE and DeepONet is comparable for cases of relatively smooth input-output mappings. However, when highly non-smooth dynamics is considered, DeepONet shows higher accuracy. We also find that for m-PCE, modest over-parameterization leads to better generalization, both within and outside of distribution, whereas aggressive over-parameterization leads to over-fitting. In contrast, an even highly over-parameterized DeepONet leads to better generalization for both smooth and non-smooth dynamics. Furthermore, we compare the performance of the above models with another operator learning model, the Fourier Neural Operator, and show that its over-parameterization also leads to better generalization. Our studies show that m-PCE can provide very good accuracy at very low training cost, whereas a highly over-parameterized DeepONet can provide better accuracy and robustness to noise but at higher training cost. In both methods, the inference cost is negligible.