# Activation Functions in Artificial Neural Networks: A Systematic Overview

arXiv: https://arxiv.org/abs/2101.09957 · HTML: https://mlsanity.com/paper/2101.09957/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 38 references that are in the map.
- First posted on arXiv 25 Jan 2021.

## Builds on (most cited first, 17 of 17)

- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning](https://mlsanity.com/paper/1702.03118/) (2017)
- [Deep Learning in Neural Networks: An Overview](https://mlsanity.com/paper/1404.7828/) (2014)
- [Self-Normalizing Neural Networks](https://mlsanity.com/paper/1706.02515/) (2017)
- [Convex Optimization: Algorithms and Complexity](https://mlsanity.com/paper/1405.4980/) (2014)
- [Empirical Evaluation of Rectified Activations in Convolutional Network](https://mlsanity.com/paper/1505.00853/) (2015)
- [Norm-Based Capacity Control in Neural Networks](https://mlsanity.com/paper/1503.00036/) (2015)
- [Maxout Networks](https://mlsanity.com/paper/1302.4389/) (2013)
- [A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks](https://mlsanity.com/paper/1810.02281/) (2018)
- [Mish: A Self Regularized Non-Monotonic Activation Function](https://mlsanity.com/paper/1908.08681/) (2019)
- [Parametric Exponential Linear Unit for Deep Convolutional Neural Networks](https://mlsanity.com/paper/1605.09332/) (2016)
- [Statistical Guarantees for Regularized Neural Networks](https://mlsanity.com/paper/2006.00294/) (2020)
- [The Quest for the Golden Activation Function](https://mlsanity.com/paper/1808.00783/) (2018)
- [Risk Bounds for Robust Deep Learning](https://mlsanity.com/paper/2009.06202/) (2020)
- [Layer Sparsity in Neural Networks](https://mlsanity.com/paper/2006.15604/) (2020)
- [A lattice-based approach to the expressivity of deep ReLU neural networks](https://mlsanity.com/paper/1902.11294/) (2019)

## Built on by (most cited first, 13 of 13)

- [Regularization and Reparameterization Avoid Vanishing Gradients in Sigmoid-Type Networks](https://mlsanity.com/paper/2106.02260/) (2021)
- [The Definitive Guide to Policy Gradients in Deep Reinforcement Learning: Theory, Algorithms and Implementations](https://mlsanity.com/paper/2401.13662/) (2024)
- [Home Electricity Data Generator (HEDGE): An open-access tool for the generation of electric vehicle, residential demand, and PV generation profiles](https://mlsanity.com/paper/2310.01661/) (2023)
- [Fermi-Dirac machines as quantizations of neurons](https://mlsanity.com/paper/2605.24386/) (2026)
- [DeepMoM: Robust Deep Learning With Median-of-Means](https://mlsanity.com/paper/2105.14035/) (2021)
- [Energy reconstruction for large liquid scintillator detectors with machine learning techniques: aggregated features approach](https://mlsanity.com/paper/2206.09040/) (2022)
- [Interpretable machine learning approach for electron antineutrino selection in a large liquid scintillator detector](https://mlsanity.com/paper/2406.12901/) (2024)
- [Targeted Deep Learning: Framework, Methods, and Applications](https://mlsanity.com/paper/2105.14052/) (2021)
- [The DeepCAR Method: Forecasting Time-Series Data That Have Change Points](https://mlsanity.com/paper/2302.11241/) (2023)
- [Nonlinearity Enhanced Adaptive Activation Functions](https://mlsanity.com/paper/2403.19896/) (2024)
- [DSReLU: A Novel Dynamic Slope Function for Superior Model Training](https://mlsanity.com/paper/2408.09156/) (2024)
- [IGLU: The Integrated Gaussian Linear Unit Activation Function](https://mlsanity.com/paper/2603.06861/) (2026)
- [The Cost of Relaxation: Evaluating the Error in Convex Neural Network Verification](https://mlsanity.com/paper/2604.18728/) (2026)

## Abstract (arXiv)

Activation functions shape the outputs of artificial neurons and, therefore, are integral parts of neural networks in general and deep learning in particular. Some activation functions, such as logistic and relu, have been used for many decades. But with deep learning becoming a mainstream research topic, new activation functions have mushroomed, leading to confusion in both theory and practice. This paper provides an analytic yet up-to-date overview of popular activation functions and their properties, which makes it a timely resource for anyone who studies or applies neural networks.