# Review and Comparison of Commonly Used Activation Functions for Deep Neural Networks

arXiv: https://arxiv.org/abs/2010.09458 · HTML: https://mlsanity.com/paper/2010.09458/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Oct 2020.

## Built on by (most cited first, 21 of 21)

- [Activation Functions in Deep Learning: A Comprehensive Survey and Benchmark](https://mlsanity.com/paper/2109.14545/) (2021)
- [Ankh: Optimized Protein Language Model Unlocks General-Purpose Modelling](https://mlsanity.com/paper/2301.06568/) (2023)
- [How important are activation functions in regression and classification? A survey, performance comparison, and future directions](https://mlsanity.com/paper/2209.02681/) (2022)
- [Development of Skip Connection in Deep Neural Networks for Computer Vision and Medical Image Analysis: A Survey](https://mlsanity.com/paper/2405.01725/) (2024)
- [Advancing Constrained Monotonic Neural Networks: Achieving Universal Approximation Beyond Bounded Activations](https://mlsanity.com/paper/2505.02537/) (2025)
- [TeLU Activation Function for Fast and Stable Deep Learning](https://mlsanity.com/paper/2412.20269/) (2024)
- [Neural ODE to model and prognose thermoacoustic instability](https://mlsanity.com/paper/2106.12758/) (2021)
- [iPINNs: Incremental learning for Physics-informed neural networks](https://mlsanity.com/paper/2304.04854/) (2023)
- [Blind Estimation of a Doubly Selective OFDM Channel: A Deep Learning Algorithm and Theory](https://mlsanity.com/paper/2206.07483/) (2022)
- [APTx: better activation function than MISH, SWISH, and ReLU's variants used in deep learning](https://mlsanity.com/paper/2209.06119/) (2022)
- [DiGRAF: Diffeomorphic Graph-Adaptive Activation Function](https://mlsanity.com/paper/2407.02013/) (2024)
- [From KAN to GR-KAN: Advancing Speech Enhancement with KAN-Based Methodology](https://mlsanity.com/paper/2412.17778/) (2024)
- [Prediction of Survival Outcomes under Clinical Presence Shift: A Joint Neural Network Architecture](https://mlsanity.com/paper/2508.05472/) (2025)
- [Spiking Neural Networks: The Future of Brain-Inspired Computing](https://mlsanity.com/paper/2510.27379/) (2025)
- [Using convolutional neural networks for stereological characterization of 3D hetero-aggregates based on synthetic STEM data](https://mlsanity.com/paper/2310.18523/) (2023)
- [Diffusing Blame: Task-Dependent Credit Assignment in Biologically Plausible Dual-Stream Networks](https://mlsanity.com/paper/2606.31700/) (2026)
- [Neuronal architecture extracts statistical temporal patterns](https://mlsanity.com/paper/2301.10203/) (2023)
- [TaLU: A Hybrid Activation Function Combining Tanh and Rectified Linear Unit to Enhance Neural Networks](https://mlsanity.com/paper/2305.04402/) (2023)
- [APTx Neuron: A Unified Trainable Neuron Architecture Integrating Activation and Computation](https://mlsanity.com/paper/2507.14270/) (2025)
- [Repair Brain Damage: Real-Numbered Error Correction Code for Neural Network](https://mlsanity.com/paper/2602.00076/) (2026)
- [A Family of Adaptive Activation Functions for Mitigating Failure Modes in Physics-Informed Neural Networks](https://mlsanity.com/paper/2603.18328/) (2026)

## Abstract (arXiv)

The primary neural networks decision-making units are activation functions. Moreover, they evaluate the output of networks neural node; thus, they are essential for the performance of the whole network. Hence, it is critical to choose the most appropriate activation function in neural networks calculation. Acharya et al. (2018) suggest that numerous recipes have been formulated over the years, though some of them are considered deprecated these days since they are unable to operate properly under some conditions. These functions have a variety of characteristics, which are deemed essential to successfully learning. Their monotonicity, individual derivatives, and finite of their range are some of these characteristics (Bach 2017). This research paper will evaluate the commonly used additive functions, such as swish, ReLU, Sigmoid, and so forth. This will be followed by their properties, own cons and pros, and particular formula application recommendations.