mlsanityData to 7 Oct 2026

Near-Minimax Optimal Estimation With Shallow ReLU Neural Networks

arXiv 2109.08844 ↗First posted 18 Sep 2021stat.ML cs.LG math.STcited by 40

Not on the map. Why?

Abstract (arXiv)

We study the problem of estimating an unknown function from noisy data using shallow ReLU neural networks. The estimators we study minimize the sum of squared data-fitting errors plus a regularization term proportional to the squared Euclidean norm of the network weights. This minimization corresponds to the common approach of training a neural network with weight decay. We quantify the performance (mean-squared error) of these neural network estimators when the data-generating function belongs to the second-order Radon-domain bounded variation space. This space of functions was recently proposed as the natural function space associated with shallow ReLU neural networks. We derive a minimax lower bound for the estimation problem for this function space and show that the neural network estimators are minimax optimal up to logarithmic factors. This minimax rate is immune to the curse of dimensionality. We quantify an explicit gap between neural networks and linear methods (which include kernel methods) by deriving a linear minimax lower bound for the estimation problem, showing that linear methods necessarily suffer the curse of dimensionality in this function space. As a result, this paper sheds light on the phenomenon that neural networks seem to break the curse of dimensionality.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Builds on 15 of its 57 references that are in the map. First posted on arXiv 18 Sep 2021.

Builds on

15 of 57 references.

References · 15

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 431
  4. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 329
  5. 2018Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionalitycited by 160
  6. 2015Path-SGD: Path-Normalized Optimization in Deep Neural Networkscited by 117
  7. 2019A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Casecited by 113
  8. 2019How do infinite width bounded norm networks look in function space?cited by 105
  9. 2021Sharp Bounds on the Approximation Rates, Metric Entropy, and n-widths of Shallow Neural Networkscited by 86
  10. 2016Approximation by Combinations of ReLU and Squared ReLU Ridge Functions with ℓ1 and ℓ0 Controlscited by 84
  11. 2021What Kinds of Functions do Deep Neural Networks Learn? Insights from Variational Spline Theorycited by 60
  12. 2018Deep Neural Networks Learn Non-Smooth Functions Effectivelycited by 57
3 more
  1. 2021Characterization of the Variation Spaces Corresponding to Shallow Neural Networkscited by 48
  2. 2019On the minimax optimality and superiority of deep neural network learning over sparse parameter spacescited by 24
  3. 2017Minimax Lower Bounds for Ridge Combinations Including Neural Netscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 4; 2023: 10; 2024: 6; 2025: 6; 2026: 12.

40 papers.

Sort:

  1. 2023Deep networks for system identification: a Surveycited by 35
  2. 2023Nonparametric regression using over-parameterized shallow ReLU neural networkscited by 24
  3. 2023Deep Learning Meets Sparse Regularization: A Signal Processing Perspectivecited by 21
  4. 2023Variation Spaces for Multi-Output Neural Networks: Insights on Multi-Task Learning and Network Compressioncited by 17
  5. 2023Optimal rates of approximation by shallow ReLUk neural networks and applications to nonparametric regressioncited by 15
  6. 2022Active Learning with Neural Networks: Insights from Nonparametric Statisticscited by 13
  7. 2022Deep Learning meets Nonparametric Regression: Are Weight-Decayed DNNs Locally Adaptive?cited by 10
  8. 2023Inductive biases of multi-task learning and finetuning: multiple regimes of feature reusecited by 9
  9. 2024Learning with Norm Constrained, Over-parameterized, Two-layer Neural Networkscited by 8
  10. 2023ReLU Neural Networks with Linear Layers are Biased Towards Single- and Multi-Index Modelscited by 7
  11. 2023Weighted variation spaces and approximation by shallow ReLU networkscited by 7
  12. 2021MARS via LASSOcited by 5
28 more
  1. 2022Intrinsic dimensionality and generalization properties of the R-norm inductive biascited by 5
  2. 2023Function-Space Optimality of Neural Architectures with Multivariate Nonlinearitiescited by 5
  3. 2024Stable Learning Using Spiking Neural Networks Equipped With Affine Encoders and Decoderscited by 5
  4. 2024ReLUs Are Sufficient for Learning Implicit Neural Representationscited by 5
  5. 2021Measuring Complexity of Learning Schemes Using Hessian-Schatten Total Variationcited by 4
  6. 2024Random ReLU Neural Networks as Non-Gaussian Processescited by 3
  7. 2025Solving Nonlinear PDEs with Sparse Radial Basis Function Networkscited by 3
  8. 2026Approximation and learning of anisotropic and mixed smooth functions by deep ReLU neural networkscited by 3
  9. 2023Integral Probability Metrics Meet Neural Networks: The Radon-Kolmogorov-Smirnov Testcited by 2
  10. 2024Nonuniform random feature models using derivative informationcited by 2
  11. 2025A Gap Between the Gaussian RKHS and Neural Networks: An Infinite-Center Asymptotic Analysiscited by 2
  12. 2025Generalization Below the Edge of Stability: The Role of Data Geometrycited by 2
  13. 2026Shallow ReLUs Networks in Lp-Type and Sobolev Spaces: Approximation and Path-Norm Controlled Generalizationcited by 2
  14. 2026Learning Sparse Compositional Functions with Norm-Constrained Neural Networkscited by 2
  15. 2026Does Sparse Connectivity Improve Generalization? Convolutional Networks Below the Edge of Stabilitycited by 1
  16. 2026A Function-Space Dichotomy for Compositional Learning: Exponential Sub-Optimality of the Neural Tangent Kernelcited by 1
  17. 2022A Regularity Theory for Static Schr"odinger Equations on ℝd in Spectral Barron Spacescited by 0
  18. 2024Dimension-independent learning rates for high-dimensional classification problemscited by 0
  19. 2025Reconstruction of frequency-localized functions from pointwise samples via least squares and deep learningcited by 0
  20. 2025Path Regularization: A Near-Complete and Optimal Nonasymptotic Generalization Theory for Multilayer Neural Networks and Double Descent Phenomenoncited by 0
  21. 2025Sobolev norm inconsistency of kernel interpolationcited by 0
  22. 2026Variation Spaces for Encoder--Decoder Neural Operators: Approximation and Generalizationcited by 0
  23. 2026Optimal Rates for Generalization of Gradient Descent Methods with Deep Neural Networkscited by 0
  24. 2026Minimax-Optimal Generalization Bounds for Smooth Deep Neural Networks Trained by (Stochastic) Gradient Descentcited by 0
  25. 2026The Information-Theoretic Benefit of Shared Representations under Orthogonality Constraintscited by 0
  26. 2026Optimal Neural Network Approximation via Empirical Least Squares with Deterministic Samplescited by 0
  27. 2026Depth Enables Local Entropy: Quadratic Depth Dependence in Deep Variation-Norm ReLU Regressioncited by 0
  28. 2026Minimax rates for learning spectral Barron functions by deep ReLU neural networkscited by 0