# Deep Network Approximation: Beyond ReLU to Diverse Activation Functions

arXiv: https://arxiv.org/abs/2307.06555 · HTML: https://mlsanity.com/paper/2307.06555/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 47 references that are in the map.
- First posted on arXiv 13 Jul 2023.

## Builds on (most cited first, 28 of 28)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning](https://mlsanity.com/paper/1702.03118/) (2017)
- [Self-Normalizing Neural Networks](https://mlsanity.com/paper/1706.02515/) (2017)
- [Error bounds for approximations with deep ReLU networks](https://mlsanity.com/paper/1610.01145/) (2016)
- [Universality of Deep Convolutional Neural Networks](https://mlsanity.com/paper/1805.10769/) (2018)
- [Optimal approximation of continuous functions by very deep ReLU networks](https://mlsanity.com/paper/1802.03620/) (2018)
- [Deep Network Approximation for Smooth Functions](https://mlsanity.com/paper/2001.03040/) (2020)
- [Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality](https://mlsanity.com/paper/1810.08033/) (2018)
- [Deep Network Approximation Characterized by Number of Neurons](https://mlsanity.com/paper/1906.05497/) (2019)
- [Optimal Approximation Rate of ReLU Networks in terms of Width and Depth](https://mlsanity.com/paper/2103.00502/) (2021)
- [Error bounds for approximations with deep ReLU neural networks in Ws,p norms](https://mlsanity.com/paper/1902.07896/) (2019)
- [Optimal Approximation with Sparsely Connected Deep Neural Networks](https://mlsanity.com/paper/1705.01714/) (2017)
- [Deep Learning via Dynamical Systems: An Approximation Perspective](https://mlsanity.com/paper/1912.10382/) (2019)
- [Continuously Differentiable Exponential Linear Units](https://mlsanity.com/paper/1704.07483/) (2017)
- [Approximation spaces of deep neural networks](https://mlsanity.com/paper/1905.01208/) (2019)
- [Error bounds for deep ReLU networks using the Kolmogorov--Arnold superposition theorem](https://mlsanity.com/paper/1906.11945/) (2019)
- [Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neurons](https://mlsanity.com/paper/2107.02397/) (2021)
- [Nonlinear Approximation via Compositions](https://mlsanity.com/paper/1902.10170/) (2019)
- [Improved Bounds on Neural Complexity for Representing Piecewise Linear Functions](https://mlsanity.com/paper/2210.07236/) (2022)
- [Approximation and Estimation for High-Dimensional Deep Learning Networks](https://mlsanity.com/paper/1809.03090/) (2018)
- [Neural Network Architecture Beyond Width and Depth](https://mlsanity.com/paper/2205.09459/) (2022)
- [Deep Network Approximation in Terms of Intrinsic Parameters](https://mlsanity.com/paper/2111.07964/) (2021)
- [Construction of neural networks for realization of localized deep learning](https://mlsanity.com/paper/1803.03503/) (2018)
- [On Enhancing Expressive Power via Compositions of Single Fixed-Size ReLU Network](https://mlsanity.com/paper/2301.12353/) (2023)

## Built on by (most cited first, 30 of 33)

- [Mixtures of Neural Operators Reduce Active Complexity in Operator Learning](https://mlsanity.com/paper/2404.09101/) (2024)
- [Non-asymptotic Approximation Error Bounds of Parameterized Quantum Circuits](https://mlsanity.com/paper/2310.07528/) (2023)
- [Don't Fear Peculiar Activation Functions: EUAF and Beyond](https://mlsanity.com/paper/2407.09580/) (2024)
- [Universal approximation results for neural networks with non-polynomial activation function over non-compact domains](https://mlsanity.com/paper/2410.14759/) (2024)
- [Deep Neural Networks with General Activations: Super-Convergence in Sobolev Norms](https://mlsanity.com/paper/2508.05141/) (2025)
- [Fourier Multi-Component and Multi-Layer Neural Networks: Unlocking High-Frequency Potential](https://mlsanity.com/paper/2502.18959/) (2025)
- [Approximation Rates in Besov Norms and Sample-Complexity of Kolmogorov-Arnold Networks with Residual Connections](https://mlsanity.com/paper/2504.15110/) (2025)
- [On the Expressive Power of Mixture-of-Experts for Structured Complex Tasks](https://mlsanity.com/paper/2505.24205/) (2025)
- [Structured and Balanced Multi-Component and Multi-Layer Neural Networks](https://mlsanity.com/paper/2407.00765/) (2024)
- [Some Super-approximation Rates of ReLU Neural Networks for Korobov Functions](https://mlsanity.com/paper/2507.10345/) (2025)
- [Learned Bayesian Cram'er-Rao Bound for Unknown Measurement Models Using Score Neural Networks](https://mlsanity.com/paper/2502.00724/) (2025)
- [Two-Dimensional Deep ReLU CNN Approximation for Korobov Functions: A Constructive Approach](https://mlsanity.com/paper/2503.07976/) (2025)
- [Beyond Universal Approximation Theorems: Algorithmic Uniform Approximation by Neural Networks Trained with Noisy Data](https://mlsanity.com/paper/2509.00924/) (2025)
- [Improvise, Adapt, Overcome -- Telescopic Adapters for Efficient Fine-tuning of Vision Language Models in Medical Imaging](https://mlsanity.com/paper/2512.13855/) (2025)
- [Multigrade Neural Network Approximation](https://mlsanity.com/paper/2601.16884/) (2026)
- [Layer-wise Geometric Approximation Rates for Deep Networks](https://mlsanity.com/paper/2604.20219/) (2026)
- [Error Analysis of Three-Layer Neural Network Trained with PGD for Deep Ritz Method](https://mlsanity.com/paper/2405.11451/) (2024)
- [Revolutionizing Communication with Deep Learning and XAI for Enhanced Arabic Sign Language Recognition](https://mlsanity.com/paper/2501.08169/) (2025)
- [Upper Approximation Bounds for Neural Oscillators](https://mlsanity.com/paper/2512.01015/) (2025)
- [Spatially Optimized Compact Deep Metric Learning Model for Similarity Search](https://mlsanity.com/paper/2404.06593/) (2024)
- [Fourier Boundary Features Network with Wider Catchers for Glass Segmentation](https://mlsanity.com/paper/2405.09459/) (2024)
- [Generative Market Equilibrium Models with Stable Adversarial Learning via Reinforcement](https://mlsanity.com/paper/2504.04300/) (2025)
- [Super-fast Rates of Convergence for Neural Network Classifiers under the Hard Margin Condition](https://mlsanity.com/paper/2505.08262/) (2025)
- [Parallel Layer Normalization for Universal Approximation](https://mlsanity.com/paper/2505.13142/) (2025)
- [Beyond Gaussian Initializations: Signal Preserving Weight Initialization for Odd-Sigmoid Activations](https://mlsanity.com/paper/2509.23085/) (2025)
- [On the Dimension-Free Approximation of Deep Neural Networks for Symmetric Korobov Functions](https://mlsanity.com/paper/2511.12398/) (2025)
- [Studying Various Activation Functions and Non-IID Data for Machine Learning Model Robustness](https://mlsanity.com/paper/2512.04264/) (2025)
- [IGLU: The Integrated Gaussian Linear Unit Activation Function](https://mlsanity.com/paper/2603.06861/) (2026)
- [A Composite Activation Function for Learning Stable Binary Representations](https://mlsanity.com/paper/2605.11558/) (2026)
- [Mitigating the Curse of Dimensionality in Uniform Convergence of Deep Neural Networks via Smooth Activations](https://mlsanity.com/paper/2606.05599/) (2026)

## Abstract (arXiv)

This paper explores the expressive power of deep neural networks for a diverse range of activation functions. An activation function set mathscrA is defined to encompass the majority of commonly used activation functions, such as mathttReLU, mathttLeakyReLU, mathttReLU2, mathttELU, mathttCELU, mathttSELU, mathttSoftplus, mathttGELU, mathttSiLU, mathttSwish, mathttMish, mathttSigmoid, mathttTanh, mathttArctan, mathttSoftsign, mathttdSiLU, and mathttSRS. We demonstrate that for any activation function varrho∈ mathscrA, a mathttReLU network of width N and depth L can be approximated to arbitrary precision by a varrho-activated network of width 3N and depth 2L on any bounded set. This finding enables the extension of most approximation results achieved with mathttReLU networks to a wide variety of other activation functions, albeit with slightly increased constants. Significantly, we establish that the (width,,depth) scaling factors can be further reduced from (3,2) to (1,1) if varrho falls within a specific subset of mathscrA. This subset includes activation functions such as mathttELU, mathttCELU, mathttSELU, mathttSoftplus, mathttGELU, mathttSiLU, mathttSwish, and mathttMish.