# Deep Neural Networks with ReLU-Sine-Exponential Activations Break Curse of Dimensionality in Approximation on H"older Class

arXiv: https://arxiv.org/abs/2103.00542 · HTML: https://mlsanity.com/paper/2103.00542/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 46 references that are in the map.
- First posted on arXiv 28 Feb 2021.

## Builds on (most cited first, 22 of 22)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Error bounds for approximations with deep ReLU networks](https://mlsanity.com/paper/1610.01145/) (2016)
- [Nonparametric regression using deep neural networks with ReLU activation function](https://mlsanity.com/paper/1708.06633/) (2017)
- [The Expressive Power of Neural Networks: A View from the Width](https://mlsanity.com/paper/1709.02540/) (2017)
- [Optimal approximation of piecewise smooth functions using deep ReLU neural networks](https://mlsanity.com/paper/1709.05289/) (2017)
- [Deep Network Approximation for Smooth Functions](https://mlsanity.com/paper/2001.03040/) (2020)
- [Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality](https://mlsanity.com/paper/1810.08033/) (2018)
- [Why and When Can Deep -- but Not Shallow -- Networks Avoid the Curse of Dimensionality: a Review](https://mlsanity.com/paper/1611.00740/) (2016)
- [Deep Network Approximation Characterized by Number of Neurons](https://mlsanity.com/paper/1906.05497/) (2019)
- [Error bounds for approximations with deep ReLU neural networks in Ws,p norms](https://mlsanity.com/paper/1902.07896/) (2019)
- [Why Deep Neural Networks for Function Approximation?](https://mlsanity.com/paper/1610.04161/) (2016)
- [Deep ReLU network approximation of functions on a manifold](https://mlsanity.com/paper/1908.00695/) (2019)
- [Neural Network Approximation: Three Hidden Layers Are Enough](https://mlsanity.com/paper/2010.14075/) (2020)
- [Provable approximation properties for deep neural networks](https://mlsanity.com/paper/1509.07385/) (2015)
- [Approximation spaces of deep neural networks](https://mlsanity.com/paper/1905.01208/) (2019)
- [Error bounds for deep ReLU networks using the Kolmogorov--Arnold superposition theorem](https://mlsanity.com/paper/1906.11945/) (2019)
- [Exponential Convergence of the Deep Neural Network Approximation for Analytic Functions](https://mlsanity.com/paper/1807.00297/) (2018)
- [Elementary superexpressive activations](https://mlsanity.com/paper/2102.10911/) (2021)
- [Deep nets for local manifold learning](https://mlsanity.com/paper/1607.07110/) (2016)
- [Adaptive Approximation and Generalization of Deep Neural Network with Intrinsic Dimensionality](https://mlsanity.com/paper/1907.02177/) (2019)
- [Towards a regularity theory for ReLU networks -- chain rule and global error estimates](https://mlsanity.com/paper/1905.04992/) (2019)
- [Deep Network with Approximation Error Being Reciprocal of Width to Power of Square Root of Depth](https://mlsanity.com/paper/2006.12231/) (2020)

## Built on by (most cited first, 7 of 7)

- [Mixtures of Neural Operators Reduce Active Complexity in Operator Learning](https://mlsanity.com/paper/2404.09101/) (2024)
- [Non-asymptotic Approximation Error Bounds of Parameterized Quantum Circuits](https://mlsanity.com/paper/2310.07528/) (2023)
- [Fourier Multi-Component and Multi-Layer Neural Networks: Unlocking High-Frequency Potential](https://mlsanity.com/paper/2502.18959/) (2025)
- [Approximation Rates in Besov Norms and Sample-Complexity of Kolmogorov-Arnold Networks with Residual Connections](https://mlsanity.com/paper/2504.15110/) (2025)
- [Approximation Rates and VC-Dimension Bounds for (P)ReLU MLP Mixture of Experts](https://mlsanity.com/paper/2402.03460/) (2024)
- [Beyond Universal Approximation Theorems: Algorithmic Uniform Approximation by Neural Networks Trained with Noisy Data](https://mlsanity.com/paper/2509.00924/) (2025)
- [Layer Separation Deep Learning Model with Auxiliary Variables for Partial Differential Equations](https://mlsanity.com/paper/2507.12766/) (2025)

## Abstract (arXiv)

In this paper, we construct neural networks with ReLU, sine and 2x as activation functions. For general continuous f defined on [0,1]d with continuity modulus ωf(·), we construct ReLU-sine-2x networks that enjoy an approximation rate O(ωf(√d)·2-M+ωfleft(frac√dNright)), where M,N∈ mathbbN+ denote the hyperparameters related to widths of the networks. As a consequence, we can construct ReLU-sine-2x network with the depth 5 and width maxleftleftlceil2d3/2left(frac3μεright)^1/αrightrceil,2leftlceillog2frac3μ dα/22εrightrceil+2right that approximates f∈ Hμα([0,1]d) within a given tolerance ε >0 measured in Lp norm p∈[1,∞), where Hμα([0,1]d) denotes the H"older continuous function class defined on [0,1]d with order α ∈ (0,1] and constant μ > 0. Therefore, the ReLU-sine-2x networks overcome the curse of dimensionality on Hμα([0,1]d). In addition to its supper expressive power, functions implemented by ReLU-sine-2x networks are (generalized) differentiable, enabling us to apply SGD to train.