mlsanityData to 14 Oct 2026

Nonparametric regression using over-parameterized shallow ReLU neural networks

arXiv 2306.08321 ↗First posted 14 Jun 2023stat.ML cs.LG math.STcited by 24

Not on the map. Why?

Abstract (arXiv)

It is shown that over-parameterized neural networks can achieve minimax optimal rates of convergence (up to logarithmic factors) for learning functions from certain smooth function classes, if the weights are suitably constrained or regularized. Specifically, we consider the nonparametric regression of estimating an unknown d-variate function by using shallow ReLU neural networks. It is assumed that the regression function is from the H"older space with smoothness α<(d+3)/2 or a variation space corresponding to shallow neural networks, which can be viewed as an infinitely wide neural network. In this setting, we prove that least squares estimators based on shallow neural networks with certain norm constraints on the weights are minimax optimal, if the network width is sufficiently large. As a byproduct, we derive a new size-independent bound for the local Rademacher complexity of shallow ReLU neural networks, which may be of independent interest.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Builds on 38 of its 66 references that are in the map. First posted on arXiv 14 Jun 2023.

Builds on

38 of 66 references.

References · 38

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  2. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  3. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  4. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  5. 2018On Lazy Training in Differentiable Programmingcited by 675
  6. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
  7. 2019Benign Overfitting in Linear Regressioncited by 549
  8. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 513
  9. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 466
  10. 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 433
  11. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 342
  12. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 331
26 more
  1. 2017Size-Independent Sample Complexity of Neural Networkscited by 290
  2. 2021Deep learning: a statistical viewpointcited by 240
  3. 2018Just Interpolate: Kernel "Ridgeless" Regression Can Generalizecited by 190
  4. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  5. 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
  6. 2018Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionalitycited by 162
  7. 2021Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolationcited by 132
  8. 2015Path-SGD: Path-Normalized Optimization in Deep Neural Networkscited by 120
  9. 2019A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Casecited by 114
  10. 2021Generalization error of random features and kernel methods: hypercontractivity and kernel matrix concentrationcited by 110
  11. 2019How do infinite width bounded norm networks look in function space?cited by 106
  12. 2018Does data interpolation contradict statistical optimality?cited by 95
  13. 2021Sharp Bounds on the Approximation Rates, Metric Entropy, and n-widths of Shallow Neural Networkscited by 86
  14. 2016Approximation by Combinations of ReLU and Squared ReLU Ridge Functions with ℓ1 and ℓ0 Controlscited by 84
  15. 2018A Priori Estimates of the Population Risk for Two-layer Neural Networkscited by 76
  16. 2019Nonparametric Regression on Low-Dimensional Manifolds using Deep ReLU Networks : Function Approximation and Statistical Recoverycited by 73
  17. 2021What Kinds of Functions do Deep Neural Networks Learn? Insights from Variational Spline Theorycited by 60
  18. 2021Understanding neural networks with reproducing kernel Banach spacescited by 50
  19. 2021Characterization of the Variation Spaces Corresponding to Shallow Neural Networkscited by 48
  20. 2018Consistency of Interpolation with Laplace Kernels is a High-Dimensional Phenomenoncited by 44
  21. 2021Near-Minimax Optimal Estimation With Shallow ReLU Neural Networkscited by 41
  22. 2021Theory of Deep Convolutional Neural Networks III: Approximating Radial Functionscited by 22
  23. 2022Approximation bounds for norm constrained neural networks with applications to regression and GANscited by 22
  24. 2023Optimal rates of approximation by shallow ReLUk neural networks and applications to nonparametric regressioncited by 15
  25. 2022Deep Learning meets Nonparametric Regression: Are Weight-Decayed DNNs Locally Adaptive?cited by 10
  26. 2019Dimension independent bounds for general shallow networkscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 7; 2025: 7; 2026: 8.

24 papers.

Sort:

  1. 2024On the rates of convergence for learning with convolutional neural networkscited by 10
  2. 2024Approximation Rates for Shallow ReLUk Neural Networks on Sobolev Spaces via the Radon Transformcited by 6
  3. 2025Transformers Can Overcome the Curse of Dimensionality: A Theoretical Study from an Approximation Perspectivecited by 5
  4. 2024Approximation Bounds for Recurrent Neural Networks with Application to Regressioncited by 4
  5. 2024Spherical Analysis of Learning Nonlinear Functionalscited by 4
  6. 2025Higher Order Approximation Rates for ReLU CNNs in Korobov Spacescited by 4
  7. 2025Approximation Bounds for Transformer Networks with Application to Regressioncited by 4
  8. 2026Learning Sparse Compositional Functions with Norm-Constrained Neural Networkscited by 3
  9. 2026Approximation and learning of anisotropic and mixed smooth functions by deep ReLU neural networkscited by 3
  10. 2024Nonuniform random feature models using derivative informationcited by 2
  11. 2026Shallow ReLUs Networks in Lp-Type and Sobolev Spaces: Approximation and Path-Norm Controlled Generalizationcited by 2
  12. 2023Barron Space for Graph Convolution Neural Networkscited by 1
12 more
  1. 2025Statistical Learning Guarantees for Group-Invariant Barron Functionscited by 1
  2. 2026Variation Spaces for Encoder--Decoder Neural Operators: Approximation and Generalizationcited by 1
  3. 2026A Function-Space Dichotomy for Compositional Learning: Exponential Sub-Optimality of the Neural Tangent Kernelcited by 1
  4. 2023The surrogate Gibbs-posterior of a corrected stochastic MALA: Towards uncertainty quantification for neural networkscited by 0
  5. 2024A least distance estimator for a multivariate regression model using deep neural networkscited by 0
  6. 2024Consistency for Large Neural Networks: Regression and Classificationcited by 0
  7. 2025Sobolev norm inconsistency of kernel interpolationcited by 0
  8. 2025Super-fast Rates of Convergence for Neural Network Classifiers under the Hard Margin Conditioncited by 0
  9. 2025ILoRA: Federated Learning with Low-Rank Adaptation for Heterogeneous Client Aggregationcited by 0
  10. 2026Towards Initialization-dependent and Non-vacuous Generalization Bounds for Overparameterized Shallow Neural Networkscited by 0
  11. 2026Error bounds in Sobolev norms for approximations with norm constrained ReLU neural networkscited by 0
  12. 2026Minimax rates for learning spectral Barron functions by deep ReLU neural networkscited by 0