# Approximation by Combinations of ReLU and Squared ReLU Ridge Functions with  ℓ1  and  ℓ0  Controls

arXiv: https://arxiv.org/abs/1607.07819 · HTML: https://mlsanity.com/paper/1607.07819/

## Key facts

- Cited by 84 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Jul 2016.

## Built on by (most cited first, 30 of 84)

- [Sharp Bounds on the Approximation Rates, Metric Entropy, and n-widths of Shallow Neural Networks](https://mlsanity.com/paper/2101.12365/) (2021)
- [Optimal Approximation Rates for Deep ReLU Neural Networks on Sobolev and Besov Spaces](https://mlsanity.com/paper/2211.14400/) (2022)
- [Characterization of the Variation Spaces Corresponding to Shallow Neural Networks](https://mlsanity.com/paper/2106.15002/) (2021)
- [Near-Minimax Optimal Estimation With Shallow ReLU Neural Networks](https://mlsanity.com/paper/2109.08844/) (2021)
- [On the expressiveness and spectral bias of KANs](https://mlsanity.com/paper/2410.01803/) (2024)
- [Random feature neural networks learn Black-Scholes type PDEs without curse of dimensionality](https://mlsanity.com/paper/2106.08900/) (2021)
- [On the Approximation Power of Two-Layer Networks of Random ReLUs](https://mlsanity.com/paper/2102.02336/) (2021)
- [Simultaneous approximation of a smooth function and its derivatives by deep neural networks with piecewise-polynomial activations](https://mlsanity.com/paper/2206.09527/) (2022)
- [Approximation Rate of the Transformer Architecture for Sequence Modeling](https://mlsanity.com/paper/2305.18475/) (2023)
- [Energy-guided Entropic Neural Optimal Transport](https://mlsanity.com/paper/2304.06094/) (2023)
- [On the Representation of Solutions to Elliptic PDEs in Barron Spaces](https://mlsanity.com/paper/2106.07539/) (2021)
- [Nonparametric regression using over-parameterized shallow ReLU neural networks](https://mlsanity.com/paper/2306.08321/) (2023)
- [Theory of Deep Convolutional Neural Networks III: Approximating Radial Functions](https://mlsanity.com/paper/2107.00896/) (2021)
- [A Priori Generalization Error Analysis of Two-Layer Neural Networks for Solving High Dimensional Schr"odinger Eigenvalue Problems](https://mlsanity.com/paper/2105.01228/) (2021)
- [A Priori Generalization Analysis of the Deep Ritz Method for Solving High Dimensional Elliptic Equations](https://mlsanity.com/paper/2101.01708/) (2021)
- [Depth separation beyond radial functions](https://mlsanity.com/paper/2102.01621/) (2021)
- [Approximation Properties of Deep ReLU CNNs](https://mlsanity.com/paper/2109.00190/) (2021)
- [Optimal rates of approximation by shallow ReLUk neural networks and applications to nonparametric regression](https://mlsanity.com/paper/2304.01561/) (2023)
- [The Expressive Power of Tuning Only the Normalization Layers](https://mlsanity.com/paper/2302.07937/) (2023)
- [On the Optimal Expressive Power of ReLU DNNs and Its Application in Approximation with Kolmogorov Superposition Theorem](https://mlsanity.com/paper/2308.05509/) (2023)
- [ReLU Deep Neural Networks from the Hierarchical Basis Perspective](https://mlsanity.com/paper/2105.04156/) (2021)
- [Active Learning for Single Neuron Models with Lipschitz Non-Linearities](https://mlsanity.com/paper/2210.13601/) (2022)
- [Rates of Approximation by ReLU Shallow Neural Networks](https://mlsanity.com/paper/2307.12461/) (2023)
- [Score-based generative models break the curse of dimensionality in learning a family of sub-Gaussian probability distributions](https://mlsanity.com/paper/2402.08082/) (2024)
- [Optimal Approximation of Zonoids and Uniform Approximation by Shallow Neural Networks](https://mlsanity.com/paper/2307.15285/) (2023)
- [Scaling Up Bayesian Uncertainty Quantification for Inverse Problems using Deep Neural Networks](https://mlsanity.com/paper/2101.03906/) (2021)
- [Max-Sliced Mutual Information](https://mlsanity.com/paper/2309.16200/) (2023)
- [Optimal learning of high-dimensional classification problems using deep neural networks](https://mlsanity.com/paper/2112.12555/) (2021)
- [Learning Ability of Interpolating Deep Convolutional Neural Networks](https://mlsanity.com/paper/2210.14184/) (2022)
- [Differentiable Neural Networks with RePU Activation: with Applications to Score Estimation and Isotonic Regression](https://mlsanity.com/paper/2305.00608/) (2023)

## Abstract (arXiv)

We establish  L∞  and  L2  error bounds for functions of many variables that are approximated by linear combinations of ReLU (rectified linear unit) and squared ReLU ridge functions with  ℓ1  and  ℓ0  controls on their inner and outer parameters. With the squared ReLU ridge function, we show that the  L2  approximation error is inversely proportional to the inner layer  ℓ0  sparsity and it need only be sublinear in the outer layer  ℓ0  sparsity. Our constructions are obtained using a variant of the Jones-Barron probabilistic method, which can be interpreted as either stratified sampling with proportionate allocation or two-stage cluster sampling. We also provide companion error lower bounds that reveal near optimality of our constructions. Despite the sparsity assumptions, we showcase the richness and flexibility of these ridge combinations by defining a large family of functions, in terms of certain spectral conditions, that are particularly well approximated by them.