# SignReLU neural network and its approximation ability

arXiv: https://arxiv.org/abs/2210.10264 · HTML: https://mlsanity.com/paper/2210.10264/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 67 references that are in the map.
- First posted on arXiv 19 Oct 2022.

## Builds on (most cited first, 26 of 26)

- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [Self-Normalizing Neural Networks](https://mlsanity.com/paper/1706.02515/) (2017)
- [Error bounds for approximations with deep ReLU networks](https://mlsanity.com/paper/1610.01145/) (2016)
- [Empirical Evaluation of Rectified Activations in Convolutional Network](https://mlsanity.com/paper/1505.00853/) (2015)
- [U2-Net: Going Deeper with Nested U-Structure for Salient Object Detection](https://mlsanity.com/paper/2005.09007/) (2020)
- [Optimal approximation of piecewise smooth functions using deep ReLU neural networks](https://mlsanity.com/paper/1709.05289/) (2017)
- [Universality of Deep Convolutional Neural Networks](https://mlsanity.com/paper/1805.10769/) (2018)
- [Deep Network Approximation for Smooth Functions](https://mlsanity.com/paper/2001.03040/) (2020)
- [Activation Functions in Deep Learning: A Comprehensive Survey and Benchmark](https://mlsanity.com/paper/2109.14545/) (2021)
- [Mish: A Self Regularized Non-Monotonic Activation Function](https://mlsanity.com/paper/1908.08681/) (2019)
- [Activation Functions: Comparison of trends in Practice and Research for Deep Learning](https://mlsanity.com/paper/1811.03378/) (2018)
- [Deep Network Approximation Characterized by Number of Neurons](https://mlsanity.com/paper/1906.05497/) (2019)
- [Error bounds for approximations with deep ReLU neural networks in Ws,p norms](https://mlsanity.com/paper/1902.07896/) (2019)
- [Optimal Approximation with Sparsely Connected Deep Neural Networks](https://mlsanity.com/paper/1705.01714/) (2017)
- [Approximation by Combinations of ReLU and Squared ReLU Ridge Functions with  ℓ1  and  ℓ0  Controls](https://mlsanity.com/paper/1607.07819/) (2016)
- [Why Deep Neural Networks for Function Approximation?](https://mlsanity.com/paper/1610.04161/) (2016)
- [Rational neural networks](https://mlsanity.com/paper/2004.01902/) (2020)
- [Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neurons](https://mlsanity.com/paper/2107.02397/) (2021)
- [Neural networks and rational functions](https://mlsanity.com/paper/1706.03301/) (2017)
- [Deep Learning-Based Video Coding: A Review and A Case Study](https://mlsanity.com/paper/1904.12462/) (2019)
- [Approximations with deep neural networks in Sobolev time-space](https://mlsanity.com/paper/2101.06115/) (2020)
- [Deep Neural Networks for Rotation-Invariance Approximation and Learning](https://mlsanity.com/paper/1904.01814/) (2019)
- [Convolutional Neural Networks for Spherical Signal Processing via Spherical Haar Tight Framelets](https://mlsanity.com/paper/2201.07890/) (2022)

## Built on by (most cited first, 4 of 4)

- [MeanSparse: Post-Training Robustness Enhancement Through Mean-Centered Feature Sparsification](https://mlsanity.com/paper/2406.05927/) (2024)
- [Convergence Analysis for Deep Sparse Coding via Convolutional Neural Networks](https://mlsanity.com/paper/2408.05540/) (2024)
- [Sparse-Aware Neural Networks for Nonlinear Functionals: Mitigating the Exponential Dependence on Dimension](https://mlsanity.com/paper/2604.06774/) (2026)
- [Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks](https://mlsanity.com/paper/2601.21242/) (2026)

## Abstract (arXiv)

Deep neural networks (DNNs) have garnered significant attention in various fields of science and technology in recent years. Activation functions define how neurons in DNNs process incoming signals for them. They are essential for learning non-linear transformations and for performing diverse computations among successive neuron layers. In the last few years, researchers have investigated the approximation ability of DNNs to explain their power and success. In this paper, we explore the approximation ability of DNNs using a different activation function, called SignReLU. Our theoretical results demonstrate that SignReLU networks outperform rational and ReLU networks in terms of approximation performance. Numerical experiments are conducted comparing SignReLU with the existing activations such as ReLU, Leaky ReLU, and ELU, which illustrate the competitive practical performance of SignReLU.