mlsanityData to 7 Oct 2026

Deep Neural Networks with ReLU-Sine-Exponential Activations Break Curse of Dimensionality in Approximation on H"older Class

arXiv 2103.00542 ↗First posted 28 Feb 2021cs.LG cs.NA cs.NEcited by 7

Not on the map. Why?

Abstract (arXiv)

In this paper, we construct neural networks with ReLU, sine and 2x as activation functions. For general continuous f defined on [0,1]d with continuity modulus ωf(·), we construct ReLU-sine-2x networks that enjoy an approximation rate O(ωf(√d)·2-M+ωfleft(frac√dNright)), where M,N∈ mathbbN+ denote the hyperparameters related to widths of the networks. As a consequence, we can construct ReLU-sine-2x network with the depth 5 and width maxleftleftlceil2d3/2left(frac3μεright)^1/αrightrceil,2leftlceillog2frac3μ dα/22εrightrceil+2right that approximates f∈ Hμα([0,1]d) within a given tolerance ε >0 measured in Lp norm p∈[1,∞), where Hμα([0,1]d) denotes the H"older continuous function class defined on [0,1]d with order α ∈ (0,1] and constant μ > 0. Therefore, the ReLU-sine-2x networks overcome the curse of dimensionality on Hμα([0,1]d). In addition to its supper expressive power, functions implemented by ReLU-sine-2x networks are (generalized) differentiable, enabling us to apply SGD to train.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 22 of its 46 references that are in the map. First posted on arXiv 28 Feb 2021.

Builds on

22 of 46 references.

References · 22

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2016Error bounds for approximations with deep ReLU networkscited by 567
  3. 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 431
  4. 2017The Expressive Power of Neural Networks: A View from the Widthcited by 318
  5. 2017Optimal approximation of piecewise smooth functions using deep ReLU neural networkscited by 176
  6. 2020Deep Network Approximation for Smooth Functionscited by 163
  7. 2018Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionalitycited by 160
  8. 2016Why and When Can Deep -- but Not Shallow -- Networks Avoid the Curse of Dimensionality: a Reviewcited by 155
  9. 2019Deep Network Approximation Characterized by Number of Neuronscited by 120
  10. 2019Error bounds for approximations with deep ReLU neural networks in Ws,p normscited by 95
  11. 2016Why Deep Neural Networks for Function Approximation?cited by 82
  12. 2019Deep ReLU network approximation of functions on a manifoldcited by 73
10 more
  1. 2020Neural Network Approximation: Three Hidden Layers Are Enoughcited by 63
  2. 2015Provable approximation properties for deep neural networkscited by 60
  3. 2019Approximation spaces of deep neural networkscited by 51
  4. 2019Error bounds for deep ReLU networks using the Kolmogorov--Arnold superposition theoremcited by 49
  5. 2018Exponential Convergence of the Deep Neural Network Approximation for Analytic Functionscited by 38
  6. 2021Elementary superexpressive activationscited by 35
  7. 2016Deep nets for local manifold learningcited by 26
  8. 2019Adaptive Approximation and Generalization of Deep Neural Network with Intrinsic Dimensionalitycited by 9
  9. 2019Towards a regularity theory for ReLU networks -- chain rule and global error estimatescited by 8
  10. 2020Deep Network with Approximation Error Being Reciprocal of Width to Power of Square Root of Depthcited by 1

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 2; 2025: 4.

7 papers.

  1. 2024Mixtures of Neural Operators Reduce Active Complexity in Operator Learningcited by 16
  2. 2023Non-asymptotic Approximation Error Bounds of Parameterized Quantum Circuitscited by 9
  3. 2025Fourier Multi-Component and Multi-Layer Neural Networks: Unlocking High-Frequency Potentialcited by 4
  4. 2024Approximation Rates and VC-Dimension Bounds for (P)ReLU MLP Mixture of Expertscited by 2
  5. 2025Approximation Rates in Besov Norms and Sample-Complexity of Kolmogorov-Arnold Networks with Residual Connectionscited by 2
  6. 2025Layer Separation Deep Learning Model with Auxiliary Variables for Partial Differential Equationscited by 1
  7. 2025Beyond Universal Approximation Theorems: Algorithmic Uniform Approximation by Neural Networks Trained with Noisy Datacited by 1