mlsanityData to 7 Oct 2026

Deep Network Approximation Characterized by Number of Neurons

arXiv 1906.05497 ↗First posted 13 Jun 2019math.NA cs.LG cs.NAcited by 120

Not on the map. Why?

Abstract (arXiv)

This paper quantitatively characterizes the approximation power of deep feed-forward neural networks (FNNs) in terms of the number of neurons. It is shown by construction that ReLU FNNs with width Obig(maxdlfloor N1/drfloor,, N+1big) and depth O(L) can approximate an arbitrary H"older continuous function of order α∈ (0,1] on [0,1]d with a nearly tight approximation rate Obig(√d N-2α/dL-2α/dbig) measured in Lp-norm for any N,L∈ mathbbN^+ and p∈[1,∞]. More generally for an arbitrary continuous function f on [0,1]d with a modulus of continuity ωf(·), the constructive approximation rate is Obig(√d,ωf( N-2/dL-2/d)big). We also extend our analysis to f on irregular domains or those localized in an varepsilon-neighborhood of a dM-dimensional smooth manifold Msubseteq [0,1]d with dMll d. Especially, in the case of an essentially low-dimensional domain, we show an approximation rate Obig(ωf(tfracvarepsilon1-δ√tfracddδ+varepsilon)+√d,ωf(tfrac√d(1-δ)√dδN-2/dδL-2/dδ)big) for ReLU FNNs to approximate f in the varepsilon-neighborhood, where dδ=Obig(dMtfracln (d/δ)δ2big) for any δ∈(0,1) as a relative error for a projection to approximate an isometry when projecting M to a dδ-dimensional domain.

Key facts. Cited by 120 papers in the map (data through 7 Oct 2026). First posted on arXiv 13 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 20; 2022: 20; 2023: 21; 2024: 20; 2025: 19; 2026: 20.

120 papers.

Sort:

By area:

  1. 2021Deep Generative Learning via Schr"odinger Bridgecited by 103
  2. 2023The Expressive Power of Low-Rank Adaptationcited by 100
  3. 2021Optimal Approximation Rate of ReLU Networks in terms of Width and Depthcited by 95
  4. 2022Deep Nonparametric Estimation of Operators between Infinite Dimensional Spacescited by 45
  5. 2021Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neuronscited by 41
  6. 2021An error analysis of generative adversarial networks for learning distributionscited by 35
  7. 2023Deep Network Approximation: Beyond ReLU to Diverse Activation Functionscited by 33
  8. 2023Nearly Optimal VC-Dimension and Pseudo-Dimension Bounds for Deep Neural Network Derivativescited by 30
  9. 2022Simultaneous approximation of a smooth function and its derivatives by deep neural networks with piecewise-polynomial activationscited by 29
  10. 2022How do noise tails impact on deep ReLU networks?cited by 25
  11. 2021Wasserstein Generative Learning of Conditional Distributioncited by 24
  12. 2022Approximation bounds for norm constrained neural networks with applications to regression and GANscited by 21
38 more
  1. 2024A Survey on Statistical Theory of Deep Learning: Approximation, Training Dynamics, and Generative Modelscited by 20
  2. 2024Auto-Train-Once: Controller Network Guided Automatic Network Pruning from Scratchcited by 20
  3. 2021Reproducing Activation Function for Deep Learningcited by 19
  4. 2021On the capacity of deep generative networks for approximating distributionscited by 19
  5. 2023Deep Operator Learning Lessens the Curse of Dimensionality for PDEscited by 18
  6. 2022Neural Network Architecture Beyond Width and Depthcited by 17
  7. 2021Least-Squares ReLU Neural Network (LSNN) Method For Linear Advection-Reaction Equationcited by 16
  8. 2021Lyapunov-Net: A Deep Neural Network Architecture for Lyapunov Function Approximationcited by 16
  9. 2022IAE-Net: Integral Autoencoders for Discretization-Invariant Learningcited by 15
  10. 2023Optimal rates of approximation by shallow ReLUk neural networks and applications to nonparametric regressioncited by 15
  11. 2021The Discovery of Dynamics via Linear Multistep Methods and Deep Learning: Error Estimationcited by 14
  12. 2021Convergence of Deep ReLU Networkscited by 14
  13. 2021Convergence of Deep Convolutional Neural Networkscited by 13
  14. 2023On the Optimal Expressive Power of ReLU DNNs and Its Application in Approximation with Kolmogorov Superposition Theoremcited by 13
  15. 2024On the optimal approximation of Sobolev and Besov functions using deep ReLU neural networkscited by 13
  16. 2021Solving PDEs on Unknown Manifolds with Machine Learningcited by 12
  17. 2021Deep Network Approximation in Terms of Intrinsic Parameterscited by 12
  18. 2021On the existence of global minima and convergence analyses for gradient descent methods in the training of deep neural networkscited by 12
  19. 2022Sample Complexity of Nonparametric Off-Policy Evaluation on Low-Dimensional Manifolds using Deep Networkscited by 12
  20. 2023Deep Nonparametric Estimation of Intrinsic Data Structures by Chart Autoencoders: Generalization Error and Robustnesscited by 10
  21. 2024On the rates of convergence for learning with convolutional neural networkscited by 10
  22. 2023Non-asymptotic Approximation Error Bounds of Parameterized Quantum Circuitscited by 9
  23. 2022Policy Evaluation for Temporal and/or Spatial Dependent Experimentscited by 8
  24. 2022Deep Neural Network Approximation of Invariant Functions through Dynamical Systemscited by 8
  25. 2023Differentiable Neural Networks with RePU Activation: with Applications to Score Estimation and Isotonic Regressioncited by 8
  26. 2023Deep Neural Networks and Finite Elements of Any Order on Arbitrary Dimensionscited by 8
  27. 2023Expressivity and Approximation Properties of Deep Neural Networks with ReLUk Activationcited by 8
  28. 2021Size and Depth Separation in Approximating Benign Functions with Neural Networkscited by 7
  29. 2021Deep Neural Networks with ReLU-Sine-Exponential Activations Break Curse of Dimensionality in Approximation on H"older Classcited by 7
  30. 2023Double-well Net for Image Segmentationcited by 7
  31. 2024Don't Fear Peculiar Activation Functions: EUAF and Beyondcited by 7
  32. 2022Benefits of Overparameterized Convolutional Residual Networks: Function Approximation under Smoothness Constraintcited by 6
  33. 2022Approximation analysis of CNNs from a feature extraction viewcited by 6
  34. 2022On the Universal Approximation Property of Deep Fully Convolutional Neural Networkscited by 6
  35. 2023On Enhancing Expressive Power via Compositions of Single Fixed-Size ReLU Networkcited by 6
  36. 2023One Neuron Saved Is One Neuron Earned: On Parametric Efficiency of Quadratic Networkscited by 6
  37. 2023Error convergence and engineering-guided hyperparameter search of PINNs: towards optimized I-FENN performancecited by 5
  38. 2023Uniform Convergence of Deep Neural Networks with Lipschitz Continuous Activation Functions and Variable Widthscited by 5

50 most cited of 120 All 120 (JSON)