# Optimal rates of approximation by shallow ReLUk neural networks and applications to nonparametric regression arXiv: https://arxiv.org/abs/2304.01561 · HTML: https://mlsanity.com/paper/2304.01561/ ## Key facts - Cited by 15 papers in the map (data through 14 Oct 2026). - Builds on 28 of its 70 references that are in the map. - First posted on arXiv 4 Apr 2023. ## Builds on (most cited first, 28 of 28) - [Spectrally-normalized margin bounds for neural networks](https://mlsanity.com/paper/1706.08498/) (2017) - [Error bounds for approximations with deep ReLU networks](https://mlsanity.com/paper/1610.01145/) (2016) - [Nonparametric regression using deep neural networks with ReLU activation function](https://mlsanity.com/paper/1708.06633/) (2017) - [Breaking the Curse of Dimensionality with Convex Neural Networks](https://mlsanity.com/paper/1412.8690/) (2014) - [A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks](https://mlsanity.com/paper/1707.09564/) (2017) - [Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networks](https://mlsanity.com/paper/1703.02930/) (2017) - [Norm-Based Capacity Control in Neural Networks](https://mlsanity.com/paper/1503.00036/) (2015) - [Size-Independent Sample Complexity of Neural Networks](https://mlsanity.com/paper/1712.06541/) (2017) - [Universality of Deep Convolutional Neural Networks](https://mlsanity.com/paper/1805.10769/) (2018) - [Optimal approximation of continuous functions by very deep ReLU networks](https://mlsanity.com/paper/1802.03620/) (2018) - [Deep Network Approximation for Smooth Functions](https://mlsanity.com/paper/2001.03040/) (2020) - [Deep Network Approximation Characterized by Number of Neurons](https://mlsanity.com/paper/1906.05497/) (2019) - [A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Case](https://mlsanity.com/paper/1910.01635/) (2019) - [How do infinite width bounded norm networks look in function space?](https://mlsanity.com/paper/1902.05040/) (2019) - [Sharp Bounds on the Approximation Rates, Metric Entropy, and n-widths of Shallow Neural Networks](https://mlsanity.com/paper/2101.12365/) (2021) - [Approximation by Combinations of ReLU and Squared ReLU Ridge Functions with ℓ1 and ℓ0 Controls](https://mlsanity.com/paper/1607.07819/) (2016) - [What Kinds of Functions do Deep Neural Networks Learn? Insights from Variational Spline Theory](https://mlsanity.com/paper/2105.03361/) (2021) - [Understanding neural networks with reproducing kernel Banach spaces](https://mlsanity.com/paper/2109.09710/) (2021) - [Characterization of the Variation Spaces Corresponding to Shallow Neural Networks](https://mlsanity.com/paper/2106.15002/) (2021) - [Near-Minimax Optimal Estimation With Shallow ReLU Neural Networks](https://mlsanity.com/paper/2109.08844/) (2021) - [Equivalence of approximation by convolutional neural networks and fully-connected networks](https://mlsanity.com/paper/1809.00973/) (2018) - [Theory of Deep Convolutional Neural Networks II: Spherical Analysis](https://mlsanity.com/paper/2007.14285/) (2020) - [Approximation bounds for norm constrained neural networks with applications to regression and GANs](https://mlsanity.com/paper/2201.09418/) (2022) - [Universal Consistency of Deep Convolutional Neural Networks](https://mlsanity.com/paper/2106.12498/) (2021) - [Approximation in shift-invariant spaces with deep ReLU neural networks](https://mlsanity.com/paper/2005.11949/) (2020) - [A general approximation lower bound in Lp norm, with applications to feed-forward neural networks](https://mlsanity.com/paper/2206.04360/) (2022) - [Learning Ability of Interpolating Deep Convolutional Neural Networks](https://mlsanity.com/paper/2210.14184/) (2022) - [Learning Distributions by Generative Adversarial Networks: Approximation and Generalization](https://mlsanity.com/paper/2205.12601/) (2022) ## Built on by (most cited first, 15 of 15) - [Nonparametric regression using over-parameterized shallow ReLU neural networks](https://mlsanity.com/paper/2306.08321/) (2023) - [On the optimal approximation of Sobolev and Besov functions using deep ReLU neural networks](https://mlsanity.com/paper/2409.00901/) (2024) - [Optimal Approximation of Zonoids and Uniform Approximation by Shallow Neural Networks](https://mlsanity.com/paper/2307.15285/) (2023) - [Expressivity and Approximation Properties of Deep Neural Networks with ReLUk Activation](https://mlsanity.com/paper/2312.16483/) (2023) - [Learning with Norm Constrained, Over-parameterized, Two-layer Neural Networks](https://mlsanity.com/paper/2404.18769/) (2024) - [Approximation Rates for Shallow ReLUk Neural Networks on Sobolev Spaces via the Radon Transform](https://mlsanity.com/paper/2408.10996/) (2024) - [Approximation Bounds for Recurrent Neural Networks with Application to Regression](https://mlsanity.com/paper/2409.05577/) (2024) - [Higher Order Approximation Rates for ReLU CNNs in Korobov Spaces](https://mlsanity.com/paper/2501.11275/) (2025) - [Approximation Bounds for Transformer Networks with Application to Regression](https://mlsanity.com/paper/2504.12175/) (2025) - [Approximation Rates and VC-Dimension Bounds for (P)ReLU MLP Mixture of Experts](https://mlsanity.com/paper/2402.03460/) (2024) - [Nonuniform random feature models using derivative information](https://mlsanity.com/paper/2410.02132/) (2024) - [Time-Frequency Analysis for Neural Networks](https://mlsanity.com/paper/2512.15992/) (2025) - [Barron Space for Graph Convolution Neural Networks](https://mlsanity.com/paper/2311.02838/) (2023) - [Error Analysis of Three-Layer Neural Network Trained with PGD for Deep Ritz Method](https://mlsanity.com/paper/2405.11451/) (2024) - [ReLU integral probability metric and its applications](https://mlsanity.com/paper/2504.18897/) (2025) ## Abstract (arXiv) We study the approximation capacity of some variation spaces corresponding to shallow ReLUk neural networks. It is shown that sufficiently smooth functions are contained in these spaces with finite variation norms. For functions with less smoothness, the approximation rates in terms of the variation norm are established. Using these results, we are able to prove the optimal approximation rates in terms of the number of neurons for shallow ReLUk neural networks. It is also shown how these results can be used to derive approximation bounds for deep neural networks and convolutional neural networks (CNNs). As applications, we study convergence rates for nonparametric regression using three ReLU neural network models: shallow neural network, over-parameterized neural network, and CNN. In particular, we show that shallow neural networks can achieve the minimax optimal rates for learning H"older functions, which complements recent results for deep neural networks. It is also proven that over-parameterized (deep or shallow) neural networks can achieve nearly optimal rates for nonparametric regression.