mlsanityData to 7 Oct 2026

SignReLU neural network and its approximation ability

arXiv 2210.10264 ↗First posted 19 Oct 2022cs.LG cs.GT eess.IVcited by 4

Not on the map. Why?

Abstract (arXiv)

Deep neural networks (DNNs) have garnered significant attention in various fields of science and technology in recent years. Activation functions define how neurons in DNNs process incoming signals for them. They are essential for learning non-linear transformations and for performing diverse computations among successive neuron layers. In the last few years, researchers have investigated the approximation ability of DNNs to explain their power and success. In this paper, we explore the approximation ability of DNNs using a different activation function, called SignReLU. Our theoretical results demonstrate that SignReLU networks outperform rational and ReLU networks in terms of approximation performance. Numerical experiments are conducted comparing SignReLU with the existing activations such as ReLU, Leaky ReLU, and ELU, which illustrate the competitive practical performance of SignReLU.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 26 of its 67 references that are in the map. First posted on arXiv 19 Oct 2022.

Builds on

26 of 67 references.

References · 26

  1. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  2. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  3. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  4. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
  5. 2017Self-Normalizing Neural Networkscited by 591
  6. 2016Error bounds for approximations with deep ReLU networkscited by 567
  7. 2015Empirical Evaluation of Rectified Activations in Convolutional Networkcited by 515
  8. 2020U2-Net: Going Deeper with Nested U-Structure for Salient Object Detectioncited by 410
  9. 2017Optimal approximation of piecewise smooth functions using deep ReLU neural networkscited by 176
  10. 2018Universality of Deep Convolutional Neural Networkscited by 176
  11. 2020Deep Network Approximation for Smooth Functionscited by 163
  12. 2021Activation Functions in Deep Learning: A Comprehensive Survey and Benchmarkcited by 153
14 more
  1. 2019Mish: A Self Regularized Non-Monotonic Activation Functioncited by 141
  2. 2018Activation Functions: Comparison of trends in Practice and Research for Deep Learningcited by 130
  3. 2019Deep Network Approximation Characterized by Number of Neuronscited by 120
  4. 2019Error bounds for approximations with deep ReLU neural networks in Ws,p normscited by 95
  5. 2017Optimal Approximation with Sparsely Connected Deep Neural Networkscited by 90
  6. 2016Approximation by Combinations of ReLU and Squared ReLU Ridge Functions with ℓ1 and ℓ0 Controlscited by 84
  7. 2016Why Deep Neural Networks for Function Approximation?cited by 82
  8. 2020Rational neural networkscited by 75
  9. 2021Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neuronscited by 41
  10. 2017Neural networks and rational functionscited by 40
  11. 2019Deep Learning-Based Video Coding: A Review and A Case Studycited by 14
  12. 2020Approximations with deep neural networks in Sobolev time-spacecited by 13
  13. 2019Deep Neural Networks for Rotation-Invariance Approximation and Learningcited by 10
  14. 2022Convolutional Neural Networks for Spherical Signal Processing via Spherical Haar Tight Frameletscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 0; 2026: 2.

4 papers.

  1. 2024MeanSparse: Post-Training Robustness Enhancement Through Mean-Centered Feature Sparsificationcited by 20
  2. 2024Convergence Analysis for Deep Sparse Coding via Convolutional Neural Networkscited by 2
  3. 2026Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networkscited by 0
  4. 2026Sparse-Aware Neural Networks for Nonlinear Functionals: Mitigating the Exponential Dependence on Dimensioncited by 0