mlsanityData to 14 Oct 2026

Expressivity and Approximation Properties of Deep Neural Networks with ReLUk Activation

arXiv 2312.16483 ↗First posted 27 Dec 2023cs.LG cs.NA cs.NEcited by 9

Not on the map. Why?

Abstract (arXiv)

In this paper, we investigate the expressivity and approximation properties of deep neural networks employing the ReLUk activation function for k ≥ 2. Although deep ReLU networks can approximate polynomials effectively, deep ReLUk networks have the capability to represent higher-degree polynomials precisely. Our initial contribution is a comprehensive, constructive proof for polynomial representation using deep ReLUk networks. This allows us to establish an upper bound on both the size and count of network parameters. Consequently, we are able to demonstrate a suboptimal approximation rate for functions from Sobolev spaces as well as for analytic functions. Additionally, through an exploration of the representation power of deep ReLUk networks for shallow networks, we reveal that deep ReLUk networks can approximate functions from a range of variation spaces, extending beyond those generated solely by the ReLUk activation function. This finding demonstrates the adaptability of deep ReLUk networks in approximating functions within various variation spaces.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 27 of its 56 references that are in the map. First posted on arXiv 27 Dec 2023.

Builds on

27 of 56 references.

References · 27

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  3. 2016Error bounds for approximations with deep ReLU networkscited by 574
  4. 2017The Expressive Power of Neural Networks: A View from the Widthcited by 320
  5. 2016Understanding Deep Neural Networks with Rectified Linear Unitscited by 257
  6. 2015The Power of Depth for Feedforward Neural Networkscited by 183
  7. 2018Universality of Deep Convolutional Neural Networkscited by 176
  8. 2016Why and When Can Deep -- but Not Shallow -- Networks Avoid the Curse of Dimensionality: a Reviewcited by 155
  9. 2019Deep Network Approximation Characterized by Number of Neuronscited by 122
  10. 2021Optimal Approximation Rate of ReLU Networks in terms of Width and Depthcited by 96
  11. 2015Representation Benefits of Deep Feedforward Networkscited by 86
  12. 2021Sharp Bounds on the Approximation Rates, Metric Entropy, and n-widths of Shallow Neural Networkscited by 86
15 more
  1. 2016Approximation by Combinations of ReLU and Squared ReLU Ridge Functions with ℓ1 and ℓ0 Controlscited by 84
  2. 2014On Using Very Large Target Vocabulary for Neural Machine Translationcited by 80
  3. 2014Question Answering with Subgraph Embeddingscited by 58
  4. 2020Theory of Deep Convolutional Neural Networks II: Spherical Analysiscited by 22
  5. 2021Theory of Deep Convolutional Neural Networks III: Approximating Radial Functionscited by 22
  6. 2021Approximation Properties of Deep ReLU CNNscited by 15
  7. 2023Optimal rates of approximation by shallow ReLUk neural networks and applications to nonparametric regressioncited by 15
  8. 2023On the Optimal Expressive Power of ReLU DNNs and Its Application in Approximation with Kolmogorov Superposition Theoremcited by 13
  9. 2021ReLU Deep Neural Networks from the Hierarchical Basis Perspectivecited by 12
  10. 2023Rates of Approximation by ReLU Shallow Neural Networkscited by 12
  11. 2013Improvements to deep convolutional neural networks for LVCSRcited by 11
  12. 2019Deep Neural Networks for Rotation-Invariance Approximation and Learningcited by 10
  13. 2023Approximation of Nonlinear Functionals Using Deep ReLU Networkscited by 8
  14. 2023Deep Neural Networks and Finite Elements of Any Order on Arbitrary Dimensionscited by 8
  15. 2023Encoding of data sets and algorithmscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 5.

9 papers.

  1. 2025Deep Neural Networks with General Activations: Super-Convergence in Sobolev Normscited by 6
  2. 2025Higher Order Approximation Rates for ReLU CNNs in Korobov Spacescited by 4
  3. 2024Variational formulation based on duality to solve partial differential equations: Use of B-splines and machine learning approximantscited by 2
  4. 2026Shallow ReLUs Networks in Lp-Type and Sobolev Spaces: Approximation and Path-Norm Controlled Generalizationcited by 2
  5. 2025Configuration-Dependent Lower Bounds for Approximation by Shallow ReLUk Networks on the Spherecited by 1
  6. 2026Do Neural Networks Really Beat the Curse of Dimensionality? A Bit-Complexity Viewcited by 1
  7. 2026Adaptive-Growth Randomized Neural Networks for Level-Set Computation of Multivalued Nonlinear First-Order PDEs with Hyperbolic Characteristicscited by 0
  8. 2026Neural Flow Operators can Approximate any Operator: Abstract Frameworks and Universal Approximationscited by 0
  9. 2026Boundary-Adapted PINNs for Elliptic Dirichlet Problems: H2(Omega) A Priori Error Bounds with Application to Mean Escape Time Computationcited by 0