# A survey on recently proposed activation functions for Deep Learning

arXiv: https://arxiv.org/abs/2204.02921 · HTML: https://mlsanity.com/paper/2204.02921/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 15 references that are in the map.
- First posted on arXiv 6 Apr 2022.

## Builds on (most cited first, 5 of 5)

- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Deep Learning in Neural Networks: An Overview](https://mlsanity.com/paper/1404.7828/) (2014)
- [A survey on modern trainable activation functions](https://mlsanity.com/paper/2005.00817/) (2020)
- [A Survey on Activation Functions and their relation with Xavier and He Normal Initialization](https://mlsanity.com/paper/2004.06632/) (2020)
- [Growing Cosine Unit: A Novel Oscillatory Activation Function That Can Speedup Training and Reduce Parameters in Convolutional Neural Networks](https://mlsanity.com/paper/2108.12943/) (2021)

## Built on by (most cited first, 6 of 6)

- [ErfReLU: Adaptive Activation Function for Deep Neural Network](https://mlsanity.com/paper/2306.01822/) (2023)
- [Automatic Grid Updates for Kolmogorov-Arnold Networks using Layer Histograms](https://mlsanity.com/paper/2511.08570/) (2025)
- [Convolutional Neural Network (CNN) to reduce construction loss in JPEG compression caused by Discrete Fourier Transform (DFT)](https://mlsanity.com/paper/2209.03475/) (2022)
- [TMSR: Tiny Multi-path CNNs for Super Resolution](https://mlsanity.com/paper/2312.01644/) (2023)
- [Nonlinearity Enhanced Adaptive Activation Functions](https://mlsanity.com/paper/2403.19896/) (2024)
- [Reward Inflation: A Healthy Stimulus for Reinforcement Learning](https://mlsanity.com/paper/2610.02545/) (2026)

## Abstract (arXiv)

Artificial neural networks (ANN), typically referred to as neural networks, are a class of Machine Learning algorithms and have achieved widespread success, having been inspired by the biological structure of the human brain. Neural networks are inherently powerful due to their ability to learn complex function approximations from data. This generalization ability has been able to impact multidisciplinary areas involving image recognition, speech recognition, natural language processing, and others. Activation functions are a crucial sub-component of neural networks. They define the output of a node in the network given a set of inputs. This survey discusses the main concepts of activation functions in neural networks, including; a brief introduction to deep neural networks, a summary of what are activation functions and how they are used in neural networks, their most common properties, the different types of activation functions, some of the challenges, limitations, and alternative solutions faced by activation functions, concluding with the final remarks.