# Nonparametric regression using over-parameterized shallow ReLU neural networks

arXiv: https://arxiv.org/abs/2306.08321 · HTML: https://mlsanity.com/paper/2306.08321/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 66 references that are in the map.
- First posted on arXiv 14 Jun 2023.

## Builds on (most cited first, 30 of 38)

- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [A Convergence Theory for Deep Learning via Over-Parameterization](https://mlsanity.com/paper/1811.03962/) (2018)
- [On Lazy Training in Differentiable Programming](https://mlsanity.com/paper/1812.07956/) (2018)
- [Gradient Descent Finds Global Minima of Deep Neural Networks](https://mlsanity.com/paper/1811.03804/) (2018)
- [Benign Overfitting in Linear Regression](https://mlsanity.com/paper/1906.11300/) (2019)
- [Surprises in High-Dimensional Ridgeless Least Squares Interpolation](https://mlsanity.com/paper/1903.08560/) (2019)
- [A Mean Field View of the Landscape of Two-Layers Neural Networks](https://mlsanity.com/paper/1804.06561/) (2018)
- [Nonparametric regression using deep neural networks with ReLU activation function](https://mlsanity.com/paper/1708.06633/) (2017)
- [In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning](https://mlsanity.com/paper/1412.6614/) (2014)
- [Breaking the Curse of Dimensionality with Convex Neural Networks](https://mlsanity.com/paper/1412.8690/) (2014)
- [Size-Independent Sample Complexity of Neural Networks](https://mlsanity.com/paper/1712.06541/) (2017)
- [Deep learning: a statistical viewpoint](https://mlsanity.com/paper/2103.09177/) (2021)
- [Just Interpolate: Kernel "Ridgeless" Regression Can Generalize](https://mlsanity.com/paper/1808.00387/) (2018)
- [Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit](https://mlsanity.com/paper/1902.06015/) (2019)
- [Towards moderate overparameterization: global convergence guarantees for training shallow neural networks](https://mlsanity.com/paper/1902.04674/) (2019)
- [Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality](https://mlsanity.com/paper/1810.08033/) (2018)
- [Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation](https://mlsanity.com/paper/2105.14368/) (2021)
- [Path-SGD: Path-Normalized Optimization in Deep Neural Networks](https://mlsanity.com/paper/1506.02617/) (2015)
- [A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Case](https://mlsanity.com/paper/1910.01635/) (2019)
- [Generalization error of random features and kernel methods: hypercontractivity and kernel matrix concentration](https://mlsanity.com/paper/2101.10588/) (2021)
- [How do infinite width bounded norm networks look in function space?](https://mlsanity.com/paper/1902.05040/) (2019)
- [Does data interpolation contradict statistical optimality?](https://mlsanity.com/paper/1806.09471/) (2018)
- [Sharp Bounds on the Approximation Rates, Metric Entropy, and n-widths of Shallow Neural Networks](https://mlsanity.com/paper/2101.12365/) (2021)
- [Approximation by Combinations of ReLU and Squared ReLU Ridge Functions with  ℓ1  and  ℓ0  Controls](https://mlsanity.com/paper/1607.07819/) (2016)
- [A Priori Estimates of the Population Risk for Two-layer Neural Networks](https://mlsanity.com/paper/1810.06397/) (2018)
- [Nonparametric Regression on Low-Dimensional Manifolds using Deep ReLU Networks : Function Approximation and Statistical Recovery](https://mlsanity.com/paper/1908.01842/) (2019)
- [What Kinds of Functions do Deep Neural Networks Learn? Insights from Variational Spline Theory](https://mlsanity.com/paper/2105.03361/) (2021)
- [Understanding neural networks with reproducing kernel Banach spaces](https://mlsanity.com/paper/2109.09710/) (2021)

## Built on by (most cited first, 24 of 24)

- [On the rates of convergence for learning with convolutional neural networks](https://mlsanity.com/paper/2403.16459/) (2024)
- [Approximation Rates for Shallow ReLUk Neural Networks on Sobolev Spaces via the Radon Transform](https://mlsanity.com/paper/2408.10996/) (2024)
- [Transformers Can Overcome the Curse of Dimensionality: A Theoretical Study from an Approximation Perspective](https://mlsanity.com/paper/2504.13558/) (2025)
- [Approximation Bounds for Recurrent Neural Networks with Application to Regression](https://mlsanity.com/paper/2409.05577/) (2024)
- [Spherical Analysis of Learning Nonlinear Functionals](https://mlsanity.com/paper/2410.01047/) (2024)
- [Higher Order Approximation Rates for ReLU CNNs in Korobov Spaces](https://mlsanity.com/paper/2501.11275/) (2025)
- [Approximation Bounds for Transformer Networks with Application to Regression](https://mlsanity.com/paper/2504.12175/) (2025)
- [Learning Sparse Compositional Functions with Norm-Constrained Neural Networks](https://mlsanity.com/paper/2605.25608/) (2026)
- [Approximation and learning of anisotropic and mixed smooth functions by deep ReLU neural networks](https://mlsanity.com/paper/2605.31152/) (2026)
- [Nonuniform random feature models using derivative information](https://mlsanity.com/paper/2410.02132/) (2024)
- [Shallow ReLUs Networks in Lp-Type and Sobolev Spaces: Approximation and Path-Norm Controlled Generalization](https://mlsanity.com/paper/2605.18468/) (2026)
- [Barron Space for Graph Convolution Neural Networks](https://mlsanity.com/paper/2311.02838/) (2023)
- [Statistical Learning Guarantees for Group-Invariant Barron Functions](https://mlsanity.com/paper/2509.23474/) (2025)
- [Variation Spaces for Encoder--Decoder Neural Operators: Approximation and Generalization](https://mlsanity.com/paper/2606.01244/) (2026)
- [A Function-Space Dichotomy for Compositional Learning: Exponential Sub-Optimality of the Neural Tangent Kernel](https://mlsanity.com/paper/2607.06382/) (2026)
- [The surrogate Gibbs-posterior of a corrected stochastic MALA: Towards uncertainty quantification for neural networks](https://mlsanity.com/paper/2310.09335/) (2023)
- [A least distance estimator for a multivariate regression model using deep neural networks](https://mlsanity.com/paper/2401.03123/) (2024)
- [Consistency for Large Neural Networks: Regression and Classification](https://mlsanity.com/paper/2409.14123/) (2024)
- [Sobolev norm inconsistency of kernel interpolation](https://mlsanity.com/paper/2504.20617/) (2025)
- [Super-fast Rates of Convergence for Neural Network Classifiers under the Hard Margin Condition](https://mlsanity.com/paper/2505.08262/) (2025)
- [ILoRA: Federated Learning with Low-Rank Adaptation for Heterogeneous Client Aggregation](https://mlsanity.com/paper/2511.16069/) (2025)
- [Towards Initialization-dependent and Non-vacuous Generalization Bounds for Overparameterized Shallow Neural Networks](https://mlsanity.com/paper/2604.00505/) (2026)
- [Error bounds in Sobolev norms for approximations with norm constrained ReLU neural networks](https://mlsanity.com/paper/2609.19937/) (2026)
- [Minimax rates for learning spectral Barron functions by deep ReLU neural networks](https://mlsanity.com/paper/2609.39020/) (2026)

## Abstract (arXiv)

It is shown that over-parameterized neural networks can achieve minimax optimal rates of convergence (up to logarithmic factors) for learning functions from certain smooth function classes, if the weights are suitably constrained or regularized. Specifically, we consider the nonparametric regression of estimating an unknown d-variate function by using shallow ReLU neural networks. It is assumed that the regression function is from the H"older space with smoothness α<(d+3)/2 or a variation space corresponding to shallow neural networks, which can be viewed as an infinitely wide neural network. In this setting, we prove that least squares estimators based on shallow neural networks with certain norm constraints on the weights are minimax optimal, if the network width is sufficiently large. As a byproduct, we derive a new size-independent bound for the local Rademacher complexity of shallow ReLU neural networks, which may be of independent interest.