# Nonlinear Approximation and (Deep) ReLU Networks

arXiv: https://arxiv.org/abs/1905.02199 · HTML: https://mlsanity.com/paper/1905.02199/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 May 2019.

## Built on by (most cited first, 30 of 45)

- [Can stable and accurate neural networks be computed? -- On the barriers of deep learning and Smale's 18th problem](https://mlsanity.com/paper/2101.08286/) (2021)
- [Random Neural Networks in the Infinite Width Limit as Gaussian Processes](https://mlsanity.com/paper/2107.01562/) (2021)
- [Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neurons](https://mlsanity.com/paper/2107.02397/) (2021)
- [When Deep Learning Meets Polyhedral Theory: A Survey](https://mlsanity.com/paper/2305.00241/) (2023)
- [On the Existence of Universal Lottery Tickets](https://mlsanity.com/paper/2111.11146/) (2021)
- [A Deep Learning approach to Reduced Order Modelling of Parameter Dependent Partial Differential Equations](https://mlsanity.com/paper/2103.06183/) (2021)
- [Neural Network Approximation of Refinable Functions](https://mlsanity.com/paper/2107.13191/) (2021)
- [Convergence of Deep Convolutional Neural Networks](https://mlsanity.com/paper/2109.13542/) (2021)
- [TransNet: Transferable Neural Networks for Partial Differential Equations](https://mlsanity.com/paper/2301.11701/) (2023)
- [On the optimal approximation of Sobolev and Besov functions using deep ReLU neural networks](https://mlsanity.com/paper/2409.00901/) (2024)
- [Sample Complexity of Nonparametric Off-Policy Evaluation on Low-Dimensional Manifolds using Deep Networks](https://mlsanity.com/paper/2206.02887/) (2022)
- [Multi-Grade Deep Learning](https://mlsanity.com/paper/2302.00150/) (2023)
- [Connections between Numerical Algorithms for PDEs and Neural Networks](https://mlsanity.com/paper/2107.14742/) (2021)
- [An Enhanced Encoder-Decoder Network Architecture for Reducing Information Loss in Image Semantic Segmentation](https://mlsanity.com/paper/2406.01605/) (2024)
- [A practical existence theorem for reduced order models based on convolutional autoencoders](https://mlsanity.com/paper/2402.00435/) (2024)
- [Stable Recovery of Entangled Weights: Towards Robust Identification of Deep Neural Networks from Minimal Samples](https://mlsanity.com/paper/2101.07150/) (2021)
- [VC dimension of Graph Neural Networks with Pfaffian activation functions](https://mlsanity.com/paper/2401.12362/) (2024)
- [High-Dimensional Distribution Generation Through Deep Neural Networks](https://mlsanity.com/paper/2107.12466/) (2021)
- [Approximation of functions with one-bit neural networks](https://mlsanity.com/paper/2112.09181/) (2021)
- [Sparse Deep Neural Network for Nonlinear Partial Differential Equations](https://mlsanity.com/paper/2207.13266/) (2022)
- [Residual Multi-Fidelity Neural Network Computing](https://mlsanity.com/paper/2310.03572/) (2023)
- [Fast and Exact Enumeration of Deep Networks Partitions Regions](https://mlsanity.com/paper/2401.11188/) (2024)
- [Deep neural network surrogates for non-smooth quantities of interest in shape uncertainty quantification](https://mlsanity.com/paper/2101.07023/) (2021)
- [Theoretical Exploration of Solutions of Feedforward ReLU Networks](https://mlsanity.com/paper/2202.01919/) (2022)
- [Approximation Power of Deep Neural Networks: an explanatory mathematical survey](https://mlsanity.com/paper/2207.09511/) (2022)
- [Successive Affine Learning for Deep Neural Networks](https://mlsanity.com/paper/2305.07996/) (2023)
- [Statistical Guarantees of Group-Invariant GANs](https://mlsanity.com/paper/2305.13517/) (2023)
- [Hypothesis Spaces for Deep Learning](https://mlsanity.com/paper/2403.03353/) (2024)
- [Designing Rotationally Invariant Neural Networks from PDEs and Variational Methods](https://mlsanity.com/paper/2108.13993/) (2021)
- [CP-PINNs: Data-Driven Changepoints Detection in PDEs Using Online Optimized Physics-Informed Neural Networks](https://mlsanity.com/paper/2208.08626/) (2022)

## Abstract (arXiv)

This article is concerned with the approximation and expressive powers of deep neural networks. This is an active research area currently producing many interesting papers. The results most commonly found in the literature prove that neural networks approximate functions with classical smoothness to the same accuracy as classical linear methods of approximation, e.g. approximation by polynomials or by piecewise polynomials on prescribed partitions. However, approximation by neural networks depending on n parameters is a form of nonlinear approximation and as such should be compared with other nonlinear methods such as variable knot splines or n-term approximation from dictionaries. The performance of neural networks in targeted applications such as machine learning indicate that they actually possess even greater approximation power than these traditional methods of nonlinear approximation. The main results of this article prove that this is indeed the case. This is done by exhibiting large classes of functions which can be efficiently captured by neural networks where classical nonlinear methods fall short of the task. The present article purposefully limits itself to studying the approximation of univariate functions by ReLU networks. Many generalizations to functions of several variables and other activation functions can be envisioned. However, even in this simplest of settings considered here, a theory that completely quantifies the approximation power of neural networks is still lacking.