mlsanityData to 14 Oct 2026

On the Optimal Expressive Power of ReLU DNNs and Its Application in Approximation with Kolmogorov Superposition Theorem

arXiv 2308.05509 ↗First posted 10 Aug 2023cs.LG cs.NA math.NAcited by 13

Not on the map. Why?

Abstract (arXiv)

This paper is devoted to studying the optimal expressive power of ReLU deep neural networks (DNNs) and its application in approximation via the Kolmogorov Superposition Theorem. We first constructively prove that any continuous piecewise linear functions on [0,1], comprising O(N2L) segments, can be represented by ReLU DNNs with L hidden layers and N neurons per layer. Subsequently, we demonstrate that this construction is optimal regarding the parameter count of the DNNs, achieved through investigating the shattering capacity of ReLU DNNs. Moreover, by invoking the Kolmogorov Superposition Theorem, we achieve an enhanced approximation rate for ReLU DNNs of arbitrary width and depth when dealing with continuous functions in high-dimensional spaces.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 35 of its 84 references that are in the map. First posted on arXiv 10 Aug 2023.

Builds on

35 of 84 references.

References · 35

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2016Wide Residual Networkscited by 3,070
  3. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  4. 2016Error bounds for approximations with deep ReLU networkscited by 574
  5. 2014On the Number of Linear Regions of Deep Neural Networkscited by 381
  6. 2017The Expressive Power of Neural Networks: A View from the Widthcited by 320
  7. 2017Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networkscited by 311
  8. 2016Understanding Deep Neural Networks with Rectified Linear Unitscited by 257
  9. 2018Optimal approximation of continuous functions by very deep ReLU networkscited by 166
  10. 2020Deep Network Approximation for Smooth Functionscited by 164
  11. 2019Deep Network Approximation Characterized by Number of Neuronscited by 122
  12. 2021Optimal Approximation Rate of ReLU Networks in terms of Width and Depthcited by 96
23 more
  1. 2019Error bounds for approximations with deep ReLU neural networks in Ws,p normscited by 95
  2. 2020The Kolmogorov-Arnold representation theorem revisitedcited by 89
  3. 2015Representation Benefits of Deep Feedforward Networkscited by 86
  4. 2021Sharp Bounds on the Approximation Rates, Metric Entropy, and n-widths of Shallow Neural Networkscited by 86
  5. 2016Approximation by Combinations of ReLU and Squared ReLU Ridge Functions with ℓ1 and ℓ0 Controlscited by 84
  6. 2018A Priori Estimates of the Population Risk for Two-layer Neural Networkscited by 76
  7. 2016Deep vs. shallow networks : An approximation theory perspectivecited by 63
  8. 2020Neural Network Approximation: Three Hidden Layers Are Enoughcited by 63
  9. 2019Error bounds for deep ReLU networks using the Kolmogorov--Arnold superposition theoremcited by 49
  10. 2021Characterization of the Variation Spaces Corresponding to Shallow Neural Networkscited by 48
  11. 2018Exponential Convergence of the Deep Neural Network Approximation for Analytic Functionscited by 38
  12. 2019Nonlinear Approximation via Compositionscited by 38
  13. 2021Elementary superexpressive activationscited by 36
  14. 2023Nearly Optimal VC-Dimension and Pseudo-Dimension Bounds for Deep Neural Network Derivativescited by 30
  15. 2020Exponential ReLU Neural Network Approximation Rates for Point and Edge Singularitiescited by 20
  16. 2020Kolmogorov Width Decay and Poor Approximators in Machine Learning: Shallow Neural Networks, Random Feature Models and Neural Tangent Kernelscited by 19
  17. 2021Reproducing Activation Function for Deep Learningcited by 19
  18. 2021On Expressivity and Trainability of Quadratic Networkscited by 19
  19. 2021ReLU Deep Neural Networks from the Hierarchical Basis Perspectivecited by 12
  20. 2022Universality and approximation bounds for echo state networks with random weightscited by 5
  21. 2022De Rham compatible Deep Neural Network FEMcited by 4
  22. 2018Bounds on the Approximation Power of Feedforward Neural Networkscited by 3
  23. 2023Sharp Lower Bounds on Interpolation by Deep ReLU Neural Networks at Irregularly Spaced Datacited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 4; 2025: 4; 2026: 2.

13 papers.

Sort:

  1. 2024KAN: Kolmogorov-Arnold Networkscited by 649
  2. 2024A comprehensive and FAIR comparison between MLP and KAN representations for differential equations and operator networkscited by 104
  3. 2024Deeper or Wider: A Perspective from Optimal Generalization Error with Sobolev Losscited by 14
  4. 2023Expressivity and Approximation Properties of Deep Neural Networks with ReLUk Activationcited by 9
  5. 2025Neural Tangent Kernel Analysis to Probe Convergence in Physics-informed Neural Solvers: PIKANs vs. PINNscited by 9
  6. 2023Deep Neural Networks and Finite Elements of Any Order on Arbitrary Dimensionscited by 8
  7. 2023Optimal Deep Neural Network Approximation for Korobov Functions with respect to Sobolev Normscited by 4
  8. 2024KAN we improve on HEP classification tasks? Kolmogorov-Arnold Networks applied to an LHC physics examplecited by 4
  9. 2025KAC: Kolmogorov-Arnold Classifier for Continual Learningcited by 4
  10. 2026Symbolic--KAN: Kolmogorov-Arnold Networks with Discrete Symbolic Structure for Interpretable Learningcited by 4
  11. 2025Some Super-approximation Rates of ReLU Neural Networks for Korobov Functionscited by 3
  12. 2025BEKAN: Boundary condition-guaranteed evolutionary Kolmogorov-Arnold networks with radial basis functions for solving PDE problemscited by 1
1 more
  1. 2026Weak-Form Evolutionary Kolmogorov-Arnold Networks for Solving Partial Differential Equationscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2022Approximation speed of quantized vs. unquantized ReLU neural networks and beyond
  2. 2025Approximation Rates in Besov Norms and Sample-Complexity of Kolmogorov-Arnold Networks with Residual Connections
  3. 2024Three Quantization Regimes for ReLU Networks
  4. 2021Error Bounds for a Matrix-Vector Product Approximation with Deep ReLU Neural Networks
  5. 2026Arbitrary-Accuracy Neural Approximation with Optimal Neuron Count and Near-Optimal Bit Complexity
  6. 2024Activation-Descent Regularization for Input Optimization of ReLU Networks
  7. 2021Initializing ReLU networks in an expressive subspace of weights
  8. 2026Wahkon: A Statistically Principled Deep RKHS Superposition Network