# Deep Network Approximation for Smooth Functions

arXiv: https://arxiv.org/abs/2001.03040 · HTML: https://mlsanity.com/paper/2001.03040/

## Key facts

- Cited by 164 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jan 2020.

## Built on by (most cited first, 30 of 164)

- [Optimal Approximation Rate of ReLU Networks in terms of Width and Depth](https://mlsanity.com/paper/2103.00502/) (2021)
- [An Overview of Diffusion Models: Applications, Guided Generation, Statistical Rates and Optimization](https://mlsanity.com/paper/2404.07771/) (2024)
- [Optimal Approximation Rates for Deep ReLU Neural Networks on Sobolev and Besov Spaces](https://mlsanity.com/paper/2211.14400/) (2022)
- [Evolutional Deep Neural Network](https://mlsanity.com/paper/2103.09959/) (2021)
- [Deep Nonparametric Estimation of Operators between Infinite Dimensional Spaces](https://mlsanity.com/paper/2201.00217/) (2022)
- [Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neurons](https://mlsanity.com/paper/2107.02397/) (2021)
- [On the expressiveness and spectral bias of KANs](https://mlsanity.com/paper/2410.01803/) (2024)
- [An error analysis of generative adversarial networks for learning distributions](https://mlsanity.com/paper/2105.13010/) (2021)
- [Symplectic Learning for Hamiltonian Neural Networks](https://mlsanity.com/paper/2106.11753/) (2021)
- [Deep Learning Methods for Partial Differential Equations and Related Parameter Identification Problems](https://mlsanity.com/paper/2212.03130/) (2022)
- [Deep Network Approximation: Beyond ReLU to Diverse Activation Functions](https://mlsanity.com/paper/2307.06555/) (2023)
- [Nearly Optimal VC-Dimension and Pseudo-Dimension Bounds for Deep Neural Network Derivatives](https://mlsanity.com/paper/2305.08466/) (2023)
- [Simultaneous approximation of a smooth function and its derivatives by deep neural networks with piecewise-polynomial activations](https://mlsanity.com/paper/2206.09527/) (2022)
- [How do noise tails impact on deep ReLU networks?](https://mlsanity.com/paper/2203.10418/) (2022)
- [Wasserstein Generative Learning of Conditional Distribution](https://mlsanity.com/paper/2112.10039/) (2021)
- [Approximation bounds for norm constrained neural networks with applications to regression and GANs](https://mlsanity.com/paper/2201.09418/) (2022)
- [Designing Universal Causal Deep Learning Models: The Case of Infinite-Dimensional Dynamical Systems from Stochastic Analysis](https://mlsanity.com/paper/2210.13300/) (2022)
- [A Survey on Statistical Theory of Deep Learning: Approximation, Training Dynamics, and Generative Models](https://mlsanity.com/paper/2401.07187/) (2024)
- [Reproducing Activation Function for Deep Learning](https://mlsanity.com/paper/2101.04844/) (2021)
- [On the capacity of deep generative networks for approximating distributions](https://mlsanity.com/paper/2101.12353/) (2021)
- [Deep Operator Learning Lessens the Curse of Dimensionality for PDEs](https://mlsanity.com/paper/2301.12227/) (2023)
- [Neural Network Architecture Beyond Width and Depth](https://mlsanity.com/paper/2205.09459/) (2022)
- [On the rate of convergence of a classifier based on a Transformer encoder](https://mlsanity.com/paper/2111.14574/) (2021)
- [IAE-Net: Integral Autoencoders for Discretization-Invariant Learning](https://mlsanity.com/paper/2203.05142/) (2022)
- [Deep Neural Networks for Nonparametric Interaction Models with Diverging Dimension](https://mlsanity.com/paper/2302.05851/) (2023)
- [Optimal rates of approximation by shallow ReLUk neural networks and applications to nonparametric regression](https://mlsanity.com/paper/2304.01561/) (2023)
- [The Discovery of Dynamics via Linear Multistep Methods and Deep Learning: Error Estimation](https://mlsanity.com/paper/2103.11488/) (2021)
- [Deeper or Wider: A Perspective from Optimal Generalization Error with Sobolev Loss](https://mlsanity.com/paper/2402.00152/) (2024)
- [Neural Scaling Laws of Deep ReLU and Deep Operator Network: A Theoretical Study](https://mlsanity.com/paper/2410.00357/) (2024)
- [Active Learning with Neural Networks: Insights from Nonparametric Statistics](https://mlsanity.com/paper/2210.08367/) (2022)

## Abstract (arXiv)

This paper establishes the (nearly) optimal approximation error characterization of deep rectified linear unit (ReLU) networks for smooth functions in terms of both width and depth simultaneously. To that end, we first prove that multivariate polynomials can be approximated by deep ReLU networks of width O(N) and depth O(L) with an approximation error O(N-L). Through local Taylor expansions and their deep ReLU network approximations, we show that deep ReLU networks of width O(Nln N) and depth O(Lln L) can approximate f∈ Cs([0,1]d) with a nearly optimal approximation error O(|f|Cs([0,1]d)N-2s/dL-2s/d). Our estimate is non-asymptotic in the sense that it is valid for arbitrary width and depth specified by N∈mathbbN^+ and L∈mathbbN^+, respectively.