mlsanityData to 14 Oct 2026

Theory-to-Practice Gap for Neural Networks and Neural Operators

arXiv 2503.18219 ↗First posted 23 Mar 2025cs.LG math.FAcited by 4

Not on the map. Why?

Abstract (arXiv)

This work studies the sampling complexity of learning with ReLU neural networks and neural operators. For mappings belonging to relevant approximation spaces, we derive upper bounds on the best-possible convergence rate of any learning algorithm, with respect to the number of samples. In the finite-dimensional case, these bounds imply a gap between the parametric and sampling complexities of learning, known as the theory-to-practice gap. In this work, a unified treatment of the theory-to-practice gap is achieved in a general Lp-setting, while at the same time improving available bounds in the literature. Furthermore, based on these results the theory-to-practice gap is extended to the infinite-dimensional setting of operator learning. Our results apply to Deep Operator Networks and integral kernel-based neural operators, including the Fourier neural operator. We show that the best-possible convergence rate in a Bochner Lp-norm is bounded by rates of order 1/p.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 16 of its 45 references that are in the map. First posted on arXiv 23 Mar 2025.

Builds on

16 of 45 references.

References · 16

  1. 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,908
  2. 2019DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operatorscited by 2,018
  3. 2016Error bounds for approximations with deep ReLU networkscited by 574
  4. 2022FourCastNet: Accelerating Global High-Resolution Weather Forecasting using Adaptive Fourier Neural Operatorscited by 188
  5. 2019Deep Neural Network Approximation Theorycited by 93
  6. 2017Optimal Approximation with Sparsely Connected Deep Neural Networkscited by 91
  7. 2024Operator Learning: Algorithms and Analysiscited by 85
  8. 2021Convergence Rates for Learning Linear Operators from Noisy Datacited by 55
  9. 2022Deep Nonparametric Estimation of Operators between Infinite Dimensional Spacescited by 45
  10. 2021Exponential Convergence of Deep Operator Networks for Elliptic Partial Differential Equationscited by 44
  11. 2023Operator learning with PCA-Net: upper and lower complexity boundscited by 44
  12. 2021Proof of the Theory-to-Practice Gap in Deep Learning via Sampling Complexity bounds for Neural Network Approximation Spacescited by 30
4 more
  1. 2024An operator learning perspective on parameter-to-observable mapscited by 22
  2. 2022Neural and spectral operator surrogates: unified construction and expression rate boundscited by 21
  3. 2024Operator Learning of Lipschitz Operators: An Information-Theoretic Perspectivecited by 11
  4. 2022Learning ReLU networks to high uniform accuracy is intractablecited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025Principled Approaches for Extending Neural Architectures to Function Spaces for Operator Learningcited by 12
  2. 2026A short tour of operator learning theory: Convergence rates, statistical limits, and open questionscited by 4
  3. 2025Learning where to learn: Training data distribution optimization for scientific machine learningcited by 3
  4. 2026The Information-Theoretic Benefit of Shared Representations under Orthogonality Constraintscited by 0