Data to 7 Oct 2026

Deep Network Approximation for Smooth Functions

arXiv 2001.03040 ↗First posted 9 Jan 2020cs.LG cs.NA math.NAcited by 163

Not on the map. Why?

Abstract (arXiv)

This paper establishes the (nearly) optimal approximation error characterization of deep rectified linear unit (ReLU) networks for smooth functions in terms of both width and depth simultaneously. To that end, we first prove that multivariate polynomials can be approximated by deep ReLU networks of width O(N) and depth O(L) with an approximation error O(N-L). Through local Taylor expansions and their deep ReLU network approximations, we show that deep ReLU networks of width O(Nln N) and depth O(Lln L) can approximate f∈ Cs([0,1]d) with a nearly optimal approximation error O(|f|Cs([0,1]d)N-2s/dL-2s/d). Our estimate is non-asymptotic in the sense that it is valid for arbitrary width and depth specified by N∈mathbbN^+ and L∈mathbbN^+, respectively.

Key facts. Cited by 163 papers in the map (data through 7 Oct 2026). First posted on arXiv 9 Jan 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 24; 2022: 25; 2023: 32; 2024: 30; 2025: 29; 2026: 23.

163 papers.

Sort:

By area:

2021Optimal Approximation Rate of ReLU Networks in terms of Width and Depthcited by 95
2024An Overview of Diffusion Models: Applications, Guided Generation, Statistical Rates and Optimizationcited by 72
2022Optimal Approximation Rates for Deep ReLU Neural Networks on Sobolev and Besov Spacescited by 50
2021Evolutional Deep Neural Networkcited by 48
2022Deep Nonparametric Estimation of Operators between Infinite Dimensional Spacescited by 45
2021Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neuronscited by 41
2024On the expressiveness and spectral bias of KANscited by 36
2021An error analysis of generative adversarial networks for learning distributionscited by 35
2021Symplectic Learning for Hamiltonian Neural Networkscited by 35
2022Deep Learning Methods for Partial Differential Equations and Related Parameter Identification Problemscited by 34
2023Deep Network Approximation: Beyond ReLU to Diverse Activation Functionscited by 33
2023Nearly Optimal VC-Dimension and Pseudo-Dimension Bounds for Deep Neural Network Derivativescited by 30

38 more

2022Simultaneous approximation of a smooth function and its derivatives by deep neural networks with piecewise-polynomial activationscited by 29
2022How do noise tails impact on deep ReLU networks?cited by 25
2021Wasserstein Generative Learning of Conditional Distributioncited by 24
2022Designing Universal Causal Deep Learning Models: The Case of Infinite-Dimensional Dynamical Systems from Stochastic Analysiscited by 22
2022Approximation bounds for norm constrained neural networks with applications to regression and GANscited by 21
2024A Survey on Statistical Theory of Deep Learning: Approximation, Training Dynamics, and Generative Modelscited by 20
2021Reproducing Activation Function for Deep Learningcited by 19
2021On the capacity of deep generative networks for approximating distributionscited by 19
2023Deep Operator Learning Lessens the Curse of Dimensionality for PDEscited by 18
2022Neural Network Architecture Beyond Width and Depthcited by 17
2021On the rate of convergence of a classifier based on a Transformer encodercited by 16
2022IAE-Net: Integral Autoencoders for Discretization-Invariant Learningcited by 15
2023Deep Neural Networks for Nonparametric Interaction Models with Diverging Dimensioncited by 15
2023Optimal rates of approximation by shallow ReLUk neural networks and applications to nonparametric regressioncited by 15
2021The Discovery of Dynamics via Linear Multistep Methods and Deep Learning: Error Estimationcited by 14
2024Deeper or Wider: A Perspective from Optimal Generalization Error with Sobolev Losscited by 14
2022Active Learning with Neural Networks: Insights from Nonparametric Statisticscited by 13
2023On the Optimal Expressive Power of ReLU DNNs and Its Application in Approximation with Kolmogorov Superposition Theoremcited by 13
2024On the optimal approximation of Sobolev and Besov functions using deep ReLU neural networkscited by 13
2024Neural Scaling Laws of Deep ReLU and Deep Operator Network: A Theoretical Studycited by 13
2021ReLU Deep Neural Networks from the Hierarchical Basis Perspectivecited by 12
2021Solving PDEs on Unknown Manifolds with Machine Learningcited by 12
2021Deep Network Approximation in Terms of Intrinsic Parameterscited by 12
2024Generalization Error Guaranteed Auto-Encoder-Based Nonlinear Model Reduction for Operator Learningcited by 12
2022DeepMed: Semiparametric Causal Mediation Analysis with Debiased Deep Learningcited by 10
2024On the rates of convergence for learning with convolutional neural networkscited by 10
2021Stationary Density Estimation of Ito Diffusions Using Deep Learningcited by 9
2021A Review of Physics-based Machine Learning in Civil Engineeringcited by 9
2021DeepParticle: learning invariant measure by a deep neural network minimizing Wasserstein distance on data generated from an interacting particle methodcited by 9
2023Non-asymptotic Approximation Error Bounds of Parameterized Quantum Circuitscited by 9
2024Automatic Differentiation is Essential in Training Neural Networks for Solving Differential Equationscited by 9
2024Posterior and variational inference for deep neural networks with heavy-tailed weightscited by 9
2022Deep Neural Network Approximation of Invariant Functions through Dynamical Systemscited by 8
2023Differentiable Neural Networks with RePU Activation: with Applications to Score Estimation and Isotonic Regressioncited by 8
2023UTOPIA: Universally Trainable Optimal Prediction Intervals Aggregationcited by 8
2023Deep Neural Networks and Finite Elements of Any Order on Arbitrary Dimensionscited by 8
2021Size and Depth Separation in Approximating Benign Functions with Neural Networkscited by 7
2021Deep Neural Networks with ReLU-Sine-Exponential Activations Break Curse of Dimensionality in Approximation on H"older Classcited by 7

50 most cited of 163 All 163 (JSON)