# ErfReLU: Adaptive Activation Function for Deep Neural Network

arXiv: https://arxiv.org/abs/2306.01822 · HTML: https://mlsanity.com/paper/2306.01822/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 28 references that are in the map.
- First posted on arXiv 2 Jun 2023.

## Builds on (most cited first, 10 of 10)

- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning](https://mlsanity.com/paper/1702.03118/) (2017)
- [Activation Functions in Deep Learning: A Comprehensive Survey and Benchmark](https://mlsanity.com/paper/2109.14545/) (2021)
- [A survey on modern trainable activation functions](https://mlsanity.com/paper/2005.00817/) (2020)
- [Discovering Parametric Activation Functions](https://mlsanity.com/paper/2006.03179/) (2020)
- [SERF: Towards better training of deep neural networks using log-Softplus ERror activation Function](https://mlsanity.com/paper/2108.09598/) (2021)
- [LiSHT: Non-Parametric Linearly Scaled Hyperbolic Tangent Activation Function for Neural Networks](https://mlsanity.com/paper/1901.05894/) (2019)
- [E-swish: Adjusting Activations to Different Network Depths](https://mlsanity.com/paper/1801.07145/) (2018)
- [A survey on recently proposed activation functions for Deep Learning](https://mlsanity.com/paper/2204.02921/) (2022)
- [ErfAct and Pserf: Non-monotonic Smooth Trainable Activation Functions](https://mlsanity.com/paper/2109.04386/) (2021)

## Built on by (most cited first, 3 of 3)

- [Trainable Highly-expressive Activation Functions](https://mlsanity.com/paper/2407.07564/) (2024)
- [Zorro: A Flexible and Differentiable Parametric Family of Activation Functions That Extends ReLU and GELU](https://mlsanity.com/paper/2409.19239/) (2024)
- [dynActivation: A Trainable Activation Family for Adaptive Nonlinearity](https://mlsanity.com/paper/2603.22154/) (2026)

## Abstract (arXiv)

Recent research has found that the activation function (AF) selected for adding non-linearity into the output can have a big impact on how effectively deep learning networks perform. Developing activation functions that can adapt simultaneously with learning is a need of time. Researchers recently started developing activation functions that can be trained throughout the learning process, known as trainable, or adaptive activation functions (AAF). Research on AAF that enhance the outcomes is still in its early stages. In this paper, a novel activation function 'ErfReLU' has been developed based on the erf function and ReLU. This function exploits the ReLU and the error function (erf) to its advantage. State of art activation functions like Sigmoid, ReLU, Tanh, and their properties have been briefly explained. Adaptive activation functions like Tanhsoft1, Tanhsoft2, Tanhsoft3, TanhLU, SAAF, ErfAct, Pserf, Smish, and Serf have also been described. Lastly, performance analysis of 9 trainable activation functions along with the proposed one namely Tanhsoft1, Tanhsoft2, Tanhsoft3, TanhLU, SAAF, ErfAct, Pserf, Smish, and Serf has been shown by applying these activation functions in MobileNet, VGG16, and ResNet models on CIFAR-10, MNIST, and FMNIST benchmark datasets.