mlsanityData to 14 Oct 2026

Approximation spaces of deep neural networks

arXiv 1905.01208 ↗First posted 3 May 2019math.FA cs.NE stat.MLcited by 51

Not on the map. Why?

Abstract (arXiv)

We study the expressivity of deep neural networks. Measuring a network's complexity by its number of connections or by its number of neurons, we consider the class of functions for which the error of best approximation with networks of a given complexity decays at a certain rate when increasing the complexity budget. Using results from classical approximation theory, we show that this class can be endowed with a (quasi)-norm that makes it a linear function space, called approximation space. We establish that allowing the networks to have certain types of "skip connections" does not change the resulting approximation spaces. We also discuss the role of the network's nonlinearity (also known as activation function) on the resulting spaces, as well as the role of depth. For the popular ReLU nonlinearity and its powers, we relate the newly constructed spaces to classical Besov spaces. The established embeddings highlight that some functions of very low Besov smoothness can nevertheless be well approximated by neural networks, if these networks are sufficiently deep.

Key facts. Cited by 51 papers in the map (data through 14 Oct 2026). First posted on arXiv 3 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 15; 2022: 10; 2023: 9; 2024: 9; 2025: 6; 2026: 2.

51 papers.

Sort:

  1. 2021Towards Lower Bounds on the Depth of ReLU Neural Networkscited by 56
  2. 2021Understanding neural networks with reproducing kernel Banach spacescited by 50
  3. 2021Universal Approximation Theorems for Differentiable Geometric Deep Learningcited by 41
  4. 2022Deep Learning Methods for Partial Differential Equations and Related Parameter Identification Problemscited by 34
  5. 2023Deep Network Approximation: Beyond ReLU to Diverse Activation Functionscited by 33
  6. 2021Proof of the Theory-to-Practice Gap in Deep Learning via Sampling Complexity bounds for Neural Network Approximation Spacescited by 30
  7. 2022Simultaneous approximation of a smooth function and its derivatives by deep neural networks with piecewise-polynomial activationscited by 29
  8. 2022Designing Universal Causal Deep Learning Models: The Geometric (Hyper)Transformercited by 22
  9. 2022Neural Network Architecture Beyond Width and Depthcited by 17
  10. 2023The Expressive Power of Tuning Only the Normalization Layerscited by 13
  11. 2021Deep Network Approximation in Terms of Intrinsic Parameterscited by 12
  12. 2021Connections between Numerical Algorithms for PDEs and Neural Networkscited by 10
38 more
  1. 2022Learning Ability of Interpolating Deep Convolutional Neural Networkscited by 9
  2. 2021Deep Neural Networks with ReLU-Sine-Exponential Activations Break Curse of Dimensionality in Approximation on H"older Classcited by 7
  3. 2021A spectral-based analysis of the separation between two-layer neural networks and linear methodscited by 7
  4. 2022Approximation results for Gradient Descent trained Shallow Neural Networks in 1dcited by 7
  5. 2023Weighted variation spaces and approximation by shallow ReLU networkscited by 7
  6. 2024Early Directional Convergence in Deep Homogeneous Neural Networks for Small Initializationscited by 7
  7. 2022Do ReLU Networks Have An Edge When Approximating Compactly-Supported Functions?cited by 6
  8. 2023On Enhancing Expressive Power via Compositions of Single Fixed-Size ReLU Networkcited by 6
  9. 2021Approximation of functions with one-bit neural networkscited by 5
  10. 2021Universal Regular Conditional Distributionscited by 4
  11. 2022Optimal bump functions for shallow ReLU networks: Weight decay, depth separation and the curse of dimensionalitycited by 4
  12. 2025Fourier Multi-Component and Multi-Layer Neural Networks: Unlocking High-Frequency Potentialcited by 4
  13. 2025Approximation Rates in Besov Norms and Sample-Complexity of Kolmogorov-Arnold Networks with Residual Connectionscited by 4
  14. 2025One model to solve them all: 2BSDE families via neural operatorscited by 4
  15. 2021Approximation Theory of Tree Tensor Networks: Tensorized Multivariate Functionscited by 3
  16. 2021Designing Rotationally Invariant Neural Networks from PDEs and Variational Methodscited by 3
  17. 2022Training Thinner and Deeper Neural Networks: Jumpstart Regularizationcited by 3
  18. 2023Universal approximation with complex-valued deep narrow neural networkscited by 3
  19. 2024Structured and Balanced Multi-Component and Multi-Layer Neural Networkscited by 3
  20. 2021Approximation properties of Residual Neural Networks for Kolmogorov PDEscited by 2
  21. 2021Integral representations of shallow neural network with Rectified Power Unit activation functioncited by 2
  22. 2022Qualitative neural network approximation over R and C: Elementary proofs for analytic and polynomial activationcited by 2
  23. 2023Operator theory, kernels, and Feedforward Neural Networkscited by 2
  24. 2023Approximation Results for Gradient Descent trained Neural Networkscited by 2
  25. 2023Approximating Langevin Monte Carlo with ResNet-like Neural Network architecturescited by 2
  26. 2024Do stable neural networks exist for classification problems? -- A new view on stability in AIcited by 2
  27. 2024Mirror Descent on Reproducing Kernel Banach Spacescited by 2
  28. 2025Low-Rank Tensor Decompositions for the Theory of Neural Networkscited by 2
  29. 2025C1-approximation with rational functions and rational neural networkscited by 2
  30. 2026Multigrade Neural Network Approximationcited by 2
  31. 2021Sobolev-type embeddings for neural network approximation spacescited by 1
  32. 2023Sampling Complexity of Deep Approximation Spacescited by 1
  33. 2024Mathematical Algorithm Design for Deep Learning under Societal and Judicial Constraints: The Algorithmic Transparency Requirementcited by 1
  34. 2024Deep Neural Networks: A Formulation Via Non-Archimedean Analysiscited by 1
  35. 2024Approximation and Gradient Descent Training with Neural Networkscited by 1
  36. 2024Theoretical Insights into CycleGAN: Analyzing Approximation and Estimation Errors in Unpaired Data Generationcited by 1
  37. 2024On the growth of the parameters of approximating ReLU neural networkscited by 0
  38. 2025Error analysis for the deep Kolmogorov methodcited by 0

50 most cited of 51 All 51 (JSON)