# Rates of Approximation by ReLU Shallow Neural Networks

arXiv: https://arxiv.org/abs/2307.12461 · HTML: https://mlsanity.com/paper/2307.12461/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 41 references that are in the map.
- First posted on arXiv 24 Jul 2023.

## Builds on (most cited first, 14 of 14)

- [Error bounds for approximations with deep ReLU networks](https://mlsanity.com/paper/1610.01145/) (2016)
- [Breaking the Curse of Dimensionality with Convex Neural Networks](https://mlsanity.com/paper/1412.8690/) (2014)
- [Optimal approximation of piecewise smooth functions using deep ReLU neural networks](https://mlsanity.com/paper/1709.05289/) (2017)
- [Universality of Deep Convolutional Neural Networks](https://mlsanity.com/paper/1805.10769/) (2018)
- [Error bounds for approximations with deep ReLU neural networks in Ws,p norms](https://mlsanity.com/paper/1902.07896/) (2019)
- [Optimal Approximation with Sparsely Connected Deep Neural Networks](https://mlsanity.com/paper/1705.01714/) (2017)
- [Approximation by Combinations of ReLU and Squared ReLU Ridge Functions with  ℓ1  and  ℓ0  Controls](https://mlsanity.com/paper/1607.07819/) (2016)
- [Deep vs. shallow networks : An approximation theory perspective](https://mlsanity.com/paper/1608.03287/) (2016)
- [Provable approximation properties for deep neural networks](https://mlsanity.com/paper/1509.07385/) (2015)
- [Theory of Deep Convolutional Neural Networks II: Spherical Analysis](https://mlsanity.com/paper/2007.14285/) (2020)
- [Theory of Deep Convolutional Neural Networks III: Approximating Radial Functions](https://mlsanity.com/paper/2107.00896/) (2021)
- [Depth Selection for Deep ReLU Nets in Feature Extraction and Generalization](https://mlsanity.com/paper/2004.00245/) (2020)
- [Function approximation with zonal function networks with activation functions analogous to the rectified linear unit functions](https://mlsanity.com/paper/1709.08174/) (2017)
- [Dimension independent bounds for general shallow networks](https://mlsanity.com/paper/1908.09880/) (2019)

## Built on by (most cited first, 12 of 12)

- [Expressivity and Approximation Properties of Deep Neural Networks with ReLUk Activation](https://mlsanity.com/paper/2312.16483/) (2023)
- [Universal approximation results for neural networks with non-polynomial activation function over non-compact domains](https://mlsanity.com/paper/2410.14759/) (2024)
- [Approximation Rates for Shallow ReLUk Neural Networks on Sobolev Spaces via the Radon Transform](https://mlsanity.com/paper/2408.10996/) (2024)
- [Approximation Bounds for Recurrent Neural Networks with Application to Regression](https://mlsanity.com/paper/2409.05577/) (2024)
- [Higher Order Approximation Rates for ReLU CNNs in Korobov Spaces](https://mlsanity.com/paper/2501.11275/) (2025)
- [On the Expressive Power of Mixture-of-Experts for Structured Complex Tasks](https://mlsanity.com/paper/2505.24205/) (2025)
- [Approximation Rates and VC-Dimension Bounds for (P)ReLU MLP Mixture of Experts](https://mlsanity.com/paper/2402.03460/) (2024)
- [Configuration-Dependent Lower Bounds for Approximation by Shallow ReLUk Networks on the Sphere](https://mlsanity.com/paper/2510.04060/) (2025)
- [Error analysis for the deep Kolmogorov method](https://mlsanity.com/paper/2508.17167/) (2025)
- [Generalization Analysis for Classification on Korobov Space](https://mlsanity.com/paper/2509.22748/) (2025)
- [Approximation of Analytic Functions by ReLU Neural Networks with Adjustable Depth and Width](https://mlsanity.com/paper/2607.10589/) (2026)
- [Shallow neural network approximation in mixed Sobolev spaces](https://mlsanity.com/paper/2609.05263/) (2026)

## Abstract (arXiv)

Neural networks activated by the rectified linear unit (ReLU) play a central role in the recent development of deep learning. The topic of approximating functions from H"older spaces by these networks is crucial for understanding the efficiency of the induced learning algorithms. Although the topic has been well investigated in the setting of deep neural networks with many layers of hidden neurons, it is still open for shallow networks having only one hidden layer. In this paper, we provide rates of uniform approximation by these networks. We show that ReLU shallow neural networks with m hidden neurons can uniformly approximate functions from the H"older space W_∞r([-1, 1]d) with rates O((log m)^frac12 +dm^-fracrdfracd+2d+4) when r<d/2 +2. Such rates are very close to the optimal one O(m^-fracrd) in the sense that fracd+2d+4 is close to 1, when the dimension d is large.