# Three Decades of Activations: A Comprehensive Survey of 400 Activation Functions for Neural Networks

arXiv: https://arxiv.org/abs/2402.09092 · HTML: https://mlsanity.com/paper/2402.09092/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Feb 2024.

## Built on by (most cited first, 23 of 23)

- [The Mean-Field Dynamics of Transformers](https://mlsanity.com/paper/2512.01868/) (2025)
- [Provably Safe Neural Network Controllers via Differential Dynamic Logic](https://mlsanity.com/paper/2402.10998/) (2024)
- [Cauchy activation function and XNet](https://mlsanity.com/paper/2409.19221/) (2024)
- [PowLU: An Activation Function for Stable Pre-Training of LLMs](https://mlsanity.com/paper/2605.25704/) (2026)
- [PReLU: Yet Another Single-Layer Solution to the XOR Problem](https://mlsanity.com/paper/2409.10821/) (2024)
- [Deep Learning as the Disciplined Construction of Tame Objects](https://mlsanity.com/paper/2509.18025/) (2025)
- [Marching Neurons: Accurate Surface Extraction for Neural Implicit Shapes](https://mlsanity.com/paper/2509.21007/) (2025)
- [DiGRAF: Diffeomorphic Graph-Adaptive Activation Function](https://mlsanity.com/paper/2407.02013/) (2024)
- [An analysis of optimization problems involving ReLU neural networks](https://mlsanity.com/paper/2502.03016/) (2025)
- [A Significantly Better Class of Activation Functions Than ReLU Like Activation Functions](https://mlsanity.com/paper/2405.04459/) (2024)
- [Activations Through Extensions: A Framework To Boost Performance Of Neural Networks](https://mlsanity.com/paper/2408.03599/) (2024)
- [Activation function optimization method: Learnable series linear units (LSLUs)](https://mlsanity.com/paper/2409.08283/) (2024)
- [Zorro: A Flexible and Differentiable Parametric Family of Activation Functions That Extends ReLU and GELU](https://mlsanity.com/paper/2409.19239/) (2024)
- [About rectified sigmoid function for enhancing the accuracy of Physics-Informed Neural Networks](https://mlsanity.com/paper/2412.20851/) (2024)
- [On the Dataless Training of Neural Networks](https://mlsanity.com/paper/2510.25962/) (2025)
- [Efficient Neural Networks with Discrete Cosine Transform Activations](https://mlsanity.com/paper/2511.03531/) (2025)
- [Brownian ReLU(Br-ReLU): A New Activation Function for a Long-Short Term Memory (LSTM) Network](https://mlsanity.com/paper/2601.16446/) (2026)
- [Evolving Multi-Channel Confidence-Aware Activation Functions for Missing Data with Channel Propagation](https://mlsanity.com/paper/2602.13864/) (2026)
- [IGLU: The Integrated Gaussian Linear Unit Activation Function](https://mlsanity.com/paper/2603.06861/) (2026)
- [Stochastic Penalty-Barrier Method for Constrained Machine Learning](https://mlsanity.com/paper/2605.18618/) (2026)
- [Activation Functions, Statistics and Learning of Higher-Order Interactions in Restricted Boltzmann Machines](https://mlsanity.com/paper/2605.19178/) (2026)
- [A Structural Interpretation of GELU and Threshold-Transmission Activations via the First-Order Loss Function](https://mlsanity.com/paper/2607.03664/) (2026)
- [When Can One Obtain Certificates of Optimality Using Positivstellensaetze?](https://mlsanity.com/paper/2609.08736/) (2026)

## Abstract (arXiv)

Neural networks have proven to be a highly effective tool for solving complex problems in many areas of life. Recently, their importance and practical usability have further been reinforced with the advent of deep learning. One of the important conditions for the success of neural networks is the choice of an appropriate activation function introducing non-linearity into the model. Many types of these functions have been proposed in the literature in the past, but there is no single comprehensive source containing their exhaustive overview. The absence of this overview, even in our experience, leads to redundancy and the unintentional rediscovery of already existing activation functions. To bridge this gap, our paper presents an extensive survey involving 400 activation functions, which is several times larger in scale than previous surveys. Our comprehensive compilation also references these surveys; however, its main goal is to provide the most comprehensive overview and systematization of previously published activation functions with links to their original sources. The secondary aim is to update the current understanding of this family of functions.