mlsanityData to 14 Oct 2026

Neural Scaling Laws of Deep ReLU and Deep Operator Network: A Theoretical Study

arXiv 2410.00357 ↗First posted 1 Oct 2024cs.LG stat.MLcited by 14

Not on the map. Why?

Abstract (arXiv)

Neural scaling laws play a pivotal role in the performance of deep neural networks and have been observed in a wide range of tasks. However, a complete theoretical framework for understanding these scaling laws remains underdeveloped. In this paper, we explore the neural scaling laws for deep operator networks, which involve learning mappings between function spaces, with a focus on the Chen and Chen style architecture. These approaches, which include the popular Deep Operator Network (DeepONet), approximate the output functions using a linear combination of learnable basis functions and coefficients that depend on the input functions. We establish a theoretical framework to quantify the neural scaling laws by analyzing its approximation and generalization errors. We articulate the relationship between the approximation and generalization errors of deep operator networks and key factors such as network model size and training data size. Moreover, we address cases where input functions exhibit low-dimensional structures, allowing us to derive tighter error bounds. These results also hold for deep ReLU networks and other similar structures. Our results offer a partial explanation of the neural scaling laws in operator learning and provide a theoretical foundation for their applications.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 35 of its 55 references that are in the map. First posted on arXiv 1 Oct 2024.

Builds on

35 of 55 references.

References · 35

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,908
  5. 2019DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operatorscited by 2,018
  6. 2020Neural Operator: Graph Kernel Network for Partial Differential Equationscited by 734
  7. 2013Speech Recognition with Deep Recurrent Neural Networkscited by 669
  8. 2017Generative Adversarial Networks: An Overviewcited by 590
  9. 2017Deep Learning Scaling is Predictable, Empiricallycited by 582
  10. 2016Error bounds for approximations with deep ReLU networkscited by 574
  11. 2017Solving high-dimensional partial differential equations using deep learningcited by 556
  12. 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 433
23 more
  1. 2021The Intrinsic Dimension of Images and Its Impact on Learningcited by 315
  2. 2020Model Reduction and Neural Networks for Parametric PDEscited by 258
  3. 2017Optimal approximation of piecewise smooth functions using deep ReLU neural networkscited by 179
  4. 2020Deep Network Approximation for Smooth Functionscited by 164
  5. 2023Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer Behaviorcited by 131
  6. 2019Nonparametric Regression on Low-Dimensional Manifolds using Deep ReLU Networks : Function Approximation and Statistical Recoverycited by 73
  7. 2018Deep Neural Networks Learn Non-Smooth Functions Effectivelycited by 57
  8. 2022Deep Nonparametric Estimation of Operators between Infinite Dimensional Spacescited by 45
  9. 2023Operator learning with PCA-Net: upper and lower complexity boundscited by 44
  10. 2019Approximation and Non-parametric Estimation of ResNet-type Convolutional Neural Networkscited by 40
  11. 2024PROSE-FD: A Multimodal PDE Foundation Model for Learning Multiple Operators for Forecasting Fluid Dynamicscited by 34
  12. 2021Besov Function Approximation and Binary Classification on Low-Dimensional Manifolds Using Convolutional Residual Networkscited by 32
  13. 2023On the training and generalization of deep operator networkscited by 31
  14. 2023The Parametric Complexity of Operator Learningcited by 28
  15. 2023Learning in latent spaces improves the predictive accuracy of deep neural operatorscited by 19
  16. 2023PROSE: Predicting Operators and Symbolic Expressions using Multimodal Transformerscited by 16
  17. 2024LeMON: Learning to Learn Multi-Operator Networkscited by 14
  18. 2023On Deep Generative Models for Approximation and Estimation of Distributions on Manifoldscited by 13
  19. 2024Generalization Error Guaranteed Auto-Encoder-Based Nonlinear Model Reduction for Operator Learningcited by 12
  20. 2023Deep Nonparametric Estimation of Intrinsic Data Structures by Chart Autoencoders: Generalization Error and Robustnesscited by 10
  21. 2024On the rates of convergence for learning with convolutional neural networkscited by 10
  22. 2024Deep Neural Networks are Adaptive to Function Regularity and Data Distribution in Approximation and Estimationcited by 3
  23. 2024WANCO: Weak Adversarial Networks for Constrained Optimization problemscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 9.

14 papers.

Sort:

  1. 2025A Deep Learning Framework for Multi-Operator Learning: Architectures and Approximation Theorycited by 9
  2. 2026A short tour of operator learning theory: Convergence rates, statistical limits, and open questionscited by 4
  3. 2026Multiple Neural Operators Achieve Near-Optimal Rates for Multi-Task Learningcited by 4
  4. 2025Coefficient-to-Basis Network: A Fine-Tunable Operator Learning Framework for Inverse Problems with Adaptive Discretizations and Theoretical Guaranteescited by 3
  5. 2025Deep Neural Operator Learning for Probabilistic Modelscited by 3
  6. 2026Generalization Bounds and Statistical Guarantees for Multi-Task and Multiple Operator Learning with MNO Networkscited by 3
  7. 2026Generalization Guarantees for Multi-Input Neural Operator Learning in Sobolev Spacescited by 3
  8. 2025Regularized Random Fourier Features and Finite Element Reconstruction for Operator Learning in Sobolev Spacecited by 2
  9. 2024DeepONet for Solving Nonlinear Partial Differential Equations with Physics-Informed Trainingcited by 1
  10. 2026MVNN: A Measure-Valued Neural Network for Learning McKean-Vlasov Dynamics from Particle Datacited by 1
  11. 2026Variation Spaces for Encoder--Decoder Neural Operators: Approximation and Generalizationcited by 1
  12. 2026Elliptic Regularity Theory in Barron Spaces and Applications to the Deep Ritz Methodcited by 1
2 more
  1. 2026Cost-Accuracy Trade-offs: Neural Operator vs Classical Numerical Solvercited by 0
  2. 2026Neural Scaling Laws of Transformer Operator Networkcited by 0