# E-swish: Adjusting Activations to Different Network Depths

arXiv: https://arxiv.org/abs/1801.07145 · HTML: https://mlsanity.com/paper/1801.07145/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Jan 2018.

## Built on by (most cited first, 6 of 6)

- [Activation Functions in Deep Learning: A Comprehensive Survey and Benchmark](https://mlsanity.com/paper/2109.14545/) (2021)
- [How important are activation functions in regression and classification? A survey, performance comparison, and future directions](https://mlsanity.com/paper/2209.02681/) (2022)
- [ErfReLU: Adaptive Activation Function for Deep Neural Network](https://mlsanity.com/paper/2306.01822/) (2023)
- [Exploring the Relationship: Transformative Adaptive Activation Functions in Comparison to Other Activation Functions](https://mlsanity.com/paper/2402.09249/) (2024)
- [Nonlinearity Enhanced Adaptive Activation Functions](https://mlsanity.com/paper/2403.19896/) (2024)
- [Swish-T : Enhancing Swish Activation with Tanh Bias for Improved Neural Network Performance](https://mlsanity.com/paper/2407.01012/) (2024)

## Abstract (arXiv)

Activation functions have a notorious impact on neural networks on both training and testing the models against the desired problem. Currently, the most used activation function is the Rectified Linear Unit (ReLU). This paper introduces a new and novel activation function, closely related with the new activation Swish = x * sigmoid(x) (Ramachandran et al., 2017) which generalizes it. We call the new activation E-swish = β x * sigmoid(x). We show that E-swish outperforms many other well-known activations including both ReLU and Swish. For example, using E-swish provided 1.5% and 4.6% accuracy improvements on Cifar10 and Cifar100 respectively for the WRN 10-2 when compared to ReLU and 0.35% and 0.6% respectively when compared to Swish. The code to reproduce all our experiments can be found at https://github.com/EricAlcaide/E-swish