# Near-Minimax Optimal Estimation With Shallow ReLU Neural Networks

arXiv: https://arxiv.org/abs/2109.08844 · HTML: https://mlsanity.com/paper/2109.08844/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 57 references that are in the map.
- First posted on arXiv 18 Sep 2021.

## Builds on (most cited first, 15 of 15)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Nonparametric regression using deep neural networks with ReLU activation function](https://mlsanity.com/paper/1708.06633/) (2017)
- [Breaking the Curse of Dimensionality with Convex Neural Networks](https://mlsanity.com/paper/1412.8690/) (2014)
- [Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality](https://mlsanity.com/paper/1810.08033/) (2018)
- [Path-SGD: Path-Normalized Optimization in Deep Neural Networks](https://mlsanity.com/paper/1506.02617/) (2015)
- [A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Case](https://mlsanity.com/paper/1910.01635/) (2019)
- [How do infinite width bounded norm networks look in function space?](https://mlsanity.com/paper/1902.05040/) (2019)
- [Sharp Bounds on the Approximation Rates, Metric Entropy, and n-widths of Shallow Neural Networks](https://mlsanity.com/paper/2101.12365/) (2021)
- [Approximation by Combinations of ReLU and Squared ReLU Ridge Functions with  ℓ1  and  ℓ0  Controls](https://mlsanity.com/paper/1607.07819/) (2016)
- [What Kinds of Functions do Deep Neural Networks Learn? Insights from Variational Spline Theory](https://mlsanity.com/paper/2105.03361/) (2021)
- [Deep Neural Networks Learn Non-Smooth Functions Effectively](https://mlsanity.com/paper/1802.04474/) (2018)
- [Characterization of the Variation Spaces Corresponding to Shallow Neural Networks](https://mlsanity.com/paper/2106.15002/) (2021)
- [On the minimax optimality and superiority of deep neural network learning over sparse parameter spaces](https://mlsanity.com/paper/1905.09195/) (2019)
- [Minimax Lower Bounds for Ridge Combinations Including Neural Nets](https://mlsanity.com/paper/1702.02828/) (2017)

## Built on by (most cited first, 30 of 41)

- [Deep networks for system identification: a Survey](https://mlsanity.com/paper/2301.12832/) (2023)
- [Nonparametric regression using over-parameterized shallow ReLU neural networks](https://mlsanity.com/paper/2306.08321/) (2023)
- [Deep Learning Meets Sparse Regularization: A Signal Processing Perspective](https://mlsanity.com/paper/2301.09554/) (2023)
- [Variation Spaces for Multi-Output Neural Networks: Insights on Multi-Task Learning and Network Compression](https://mlsanity.com/paper/2305.16534/) (2023)
- [Optimal rates of approximation by shallow ReLUk neural networks and applications to nonparametric regression](https://mlsanity.com/paper/2304.01561/) (2023)
- [Active Learning with Neural Networks: Insights from Nonparametric Statistics](https://mlsanity.com/paper/2210.08367/) (2022)
- [Deep Learning meets Nonparametric Regression: Are Weight-Decayed DNNs Locally Adaptive?](https://mlsanity.com/paper/2204.09664/) (2022)
- [Inductive biases of multi-task learning and finetuning: multiple regimes of feature reuse](https://mlsanity.com/paper/2310.02396/) (2023)
- [ReLU Neural Networks with Linear Layers are Biased Towards Single- and Multi-Index Models](https://mlsanity.com/paper/2305.15598/) (2023)
- [Learning with Norm Constrained, Over-parameterized, Two-layer Neural Networks](https://mlsanity.com/paper/2404.18769/) (2024)
- [Weighted variation spaces and approximation by shallow ReLU networks](https://mlsanity.com/paper/2307.15772/) (2023)
- [MARS via LASSO](https://mlsanity.com/paper/2111.11694/) (2021)
- [Intrinsic dimensionality and generalization properties of the R-norm inductive bias](https://mlsanity.com/paper/2206.05317/) (2022)
- [Function-Space Optimality of Neural Architectures with Multivariate Nonlinearities](https://mlsanity.com/paper/2310.03696/) (2023)
- [Stable Learning Using Spiking Neural Networks Equipped With Affine Encoders and Decoders](https://mlsanity.com/paper/2404.04549/) (2024)
- [ReLUs Are Sufficient for Learning Implicit Neural Representations](https://mlsanity.com/paper/2406.02529/) (2024)
- [Measuring Complexity of Learning Schemes Using Hessian-Schatten Total Variation](https://mlsanity.com/paper/2112.06209/) (2021)
- [Random ReLU Neural Networks as Non-Gaussian Processes](https://mlsanity.com/paper/2405.10229/) (2024)
- [Solving Nonlinear PDEs with Sparse Radial Basis Function Networks](https://mlsanity.com/paper/2505.07765/) (2025)
- [Generalization Below the Edge of Stability: The Role of Data Geometry](https://mlsanity.com/paper/2510.18120/) (2025)
- [Learning Sparse Compositional Functions with Norm-Constrained Neural Networks](https://mlsanity.com/paper/2605.25608/) (2026)
- [Approximation and learning of anisotropic and mixed smooth functions by deep ReLU neural networks](https://mlsanity.com/paper/2605.31152/) (2026)
- [Integral Probability Metrics Meet Neural Networks: The Radon-Kolmogorov-Smirnov Test](https://mlsanity.com/paper/2309.02422/) (2023)
- [Nonuniform random feature models using derivative information](https://mlsanity.com/paper/2410.02132/) (2024)
- [A Gap Between the Gaussian RKHS and Neural Networks: An Infinite-Center Asymptotic Analysis](https://mlsanity.com/paper/2502.16331/) (2025)
- [Shallow ReLUs Networks in Lp-Type and Sobolev Spaces: Approximation and Path-Norm Controlled Generalization](https://mlsanity.com/paper/2605.18468/) (2026)
- [Does Sparse Connectivity Improve Generalization? Convolutional Networks Below the Edge of Stability](https://mlsanity.com/paper/2603.04807/) (2026)
- [Variation Spaces for Encoder--Decoder Neural Operators: Approximation and Generalization](https://mlsanity.com/paper/2606.01244/) (2026)
- [A Function-Space Dichotomy for Compositional Learning: Exponential Sub-Optimality of the Neural Tangent Kernel](https://mlsanity.com/paper/2607.06382/) (2026)
- [A Regularity Theory for Static Schr"odinger Equations on ℝd in Spectral Barron Spaces](https://mlsanity.com/paper/2201.10072/) (2022)

## Abstract (arXiv)

We study the problem of estimating an unknown function from noisy data using shallow ReLU neural networks. The estimators we study minimize the sum of squared data-fitting errors plus a regularization term proportional to the squared Euclidean norm of the network weights. This minimization corresponds to the common approach of training a neural network with weight decay. We quantify the performance (mean-squared error) of these neural network estimators when the data-generating function belongs to the second-order Radon-domain bounded variation space. This space of functions was recently proposed as the natural function space associated with shallow ReLU neural networks. We derive a minimax lower bound for the estimation problem for this function space and show that the neural network estimators are minimax optimal up to logarithmic factors. This minimax rate is immune to the curse of dimensionality. We quantify an explicit gap between neural networks and linear methods (which include kernel methods) by deriving a linear minimax lower bound for the estimation problem, showing that linear methods necessarily suffer the curse of dimensionality in this function space. As a result, this paper sheds light on the phenomenon that neural networks seem to break the curse of dimensionality.