mlsanityData to 7 Oct 2026

Deep Network Approximation: Beyond ReLU to Diverse Activation Functions

arXiv 2307.06555 ↗First posted 13 Jul 2023cs.LG stat.MLcited by 33

Not on the map. Why?

Abstract (arXiv)

This paper explores the expressive power of deep neural networks for a diverse range of activation functions. An activation function set mathscrA is defined to encompass the majority of commonly used activation functions, such as mathttReLU, mathttLeakyReLU, mathttReLU2, mathttELU, mathttCELU, mathttSELU, mathttSoftplus, mathttGELU, mathttSiLU, mathttSwish, mathttMish, mathttSigmoid, mathttTanh, mathttArctan, mathttSoftsign, mathttdSiLU, and mathttSRS. We demonstrate that for any activation function varrho∈ mathscrA, a mathttReLU network of width N and depth L can be approximated to arbitrary precision by a varrho-activated network of width 3N and depth 2L on any bounded set. This finding enables the extension of most approximation results achieved with mathttReLU networks to a wide variety of other activation functions, albeit with slightly increased constants. Significantly, we establish that the (width,,depth) scaling factors can be further reduced from (3,2) to (1,1) if varrho falls within a specific subset of mathscrA. This subset includes activation functions such as mathttELU, mathttCELU, mathttSELU, mathttSoftplus, mathttGELU, mathttSiLU, mathttSwish, and mathttMish.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Builds on 28 of its 47 references that are in the map. First posted on arXiv 13 Jul 2023.

Builds on

28 of 47 references.

References · 28

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  4. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  5. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
  6. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 935
  7. 2017Self-Normalizing Neural Networkscited by 591
  8. 2016Error bounds for approximations with deep ReLU networkscited by 567
  9. 2018Universality of Deep Convolutional Neural Networkscited by 176
  10. 2018Optimal approximation of continuous functions by very deep ReLU networkscited by 165
  11. 2020Deep Network Approximation for Smooth Functionscited by 163
  12. 2018Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionalitycited by 160
16 more
  1. 2019Deep Network Approximation Characterized by Number of Neuronscited by 120
  2. 2019Error bounds for approximations with deep ReLU neural networks in Ws,p normscited by 95
  3. 2021Optimal Approximation Rate of ReLU Networks in terms of Width and Depthcited by 95
  4. 2017Optimal Approximation with Sparsely Connected Deep Neural Networkscited by 90
  5. 2019Deep Learning via Dynamical Systems: An Approximation Perspectivecited by 78
  6. 2017Continuously Differentiable Exponential Linear Unitscited by 73
  7. 2019Approximation spaces of deep neural networkscited by 51
  8. 2019Error bounds for deep ReLU networks using the Kolmogorov--Arnold superposition theoremcited by 49
  9. 2021Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neuronscited by 41
  10. 2019Nonlinear Approximation via Compositionscited by 37
  11. 2022Improved Bounds on Neural Complexity for Representing Piecewise Linear Functionscited by 26
  12. 2018Approximation and Estimation for High-Dimensional Deep Learning Networkscited by 22
  13. 2022Neural Network Architecture Beyond Width and Depthcited by 17
  14. 2021Deep Network Approximation in Terms of Intrinsic Parameterscited by 12
  15. 2018Construction of neural networks for realization of localized deep learningcited by 7
  16. 2023On Enhancing Expressive Power via Compositions of Single Fixed-Size ReLU Networkcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 7; 2025: 17; 2026: 8.

33 papers.

Sort:

By area:

  1. 2024Mixtures of Neural Operators Reduce Active Complexity in Operator Learningcited by 16
  2. 2023Non-asymptotic Approximation Error Bounds of Parameterized Quantum Circuitscited by 9
  3. 2024Don't Fear Peculiar Activation Functions: EUAF and Beyondcited by 7
  4. 2024Universal approximation results for neural networks with non-polynomial activation function over non-compact domainscited by 7
  5. 2025Deep Neural Networks with General Activations: Super-Convergence in Sobolev Normscited by 5
  6. 2025Fourier Multi-Component and Multi-Layer Neural Networks: Unlocking High-Frequency Potentialcited by 4
  7. 2025On the Expressive Power of Mixture-of-Experts for Structured Complex Taskscited by 4
  8. 2024Structured and Balanced Multi-Component and Multi-Layer Neural Networkscited by 3
  9. 2025Some Super-approximation Rates of ReLU Neural Networks for Korobov Functionscited by 3
  10. 2025Two-Dimensional Deep ReLU CNN Approximation for Korobov Functions: A Constructive Approachcited by 2
  11. 2025Approximation Rates in Besov Norms and Sample-Complexity of Kolmogorov-Arnold Networks with Residual Connectionscited by 2
  12. 2024Error Analysis of Three-Layer Neural Network Trained with PGD for Deep Ritz Methodcited by 1
21 more
  1. 2025Revolutionizing Communication with Deep Learning and XAI for Enhanced Arabic Sign Language Recognitioncited by 1
  2. 2025Beyond Universal Approximation Theorems: Algorithmic Uniform Approximation by Neural Networks Trained with Noisy Datacited by 1
  3. 2025Upper Approximation Bounds for Neural Oscillatorscited by 1
  4. 2025Improvise, Adapt, Overcome -- Telescopic Adapters for Efficient Fine-tuning of Vision Language Models in Medical Imagingcited by 1
  5. 2026Multigrade Neural Network Approximationcited by 1
  6. 2026Layer-wise Geometric Approximation Rates for Deep Networkscited by 1
  7. 2024Spatially Optimized Compact Deep Metric Learning Model for Similarity Searchcited by 0
  8. 2024Fourier Boundary Features Network with Wider Catchers for Glass Segmentationcited by 0
  9. 2025Learned Bayesian Cram'er-Rao Bound for Unknown Measurement Models Using Score Neural Networkscited by 0
  10. 2025Generative Market Equilibrium Models with Stable Adversarial Learning via Reinforcementcited by 0
  11. 2025Super-fast Rates of Convergence for Neural Network Classifiers under the Hard Margin Conditioncited by 0
  12. 2025Parallel Layer Normalization for Universal Approximationcited by 0
  13. 2025Beyond Gaussian Initializations: Signal Preserving Weight Initialization for Odd-Sigmoid Activationscited by 0
  14. 2025On the Dimension-Free Approximation of Deep Neural Networks for Symmetric Korobov Functionscited by 0
  15. 2025Studying Various Activation Functions and Non-IID Data for Machine Learning Model Robustnesscited by 0
  16. 2026IGLU: The Integrated Gaussian Linear Unit Activation Functioncited by 0
  17. 2026A Composite Activation Function for Learning Stable Binary Representationscited by 0
  18. 2026Mitigating the Curse of Dimensionality in Uniform Convergence of Deep Neural Networks via Smooth Activationscited by 0
  19. 2026Approximation of Analytic Functions by ReLU Neural Networks with Adjustable Depth and Widthcited by 0
  20. 2026NeuralChaos: Optimal Adapted Approximation of Square Integrable Predictable Processescited by 0
  21. 2026On the Applicability of Safety Nets: A Safety-By-Design Solution for Certifying Neural Networkscited by 0