mlsanityData to 14 Oct 2026

Activation Functions in Artificial Neural Networks: A Systematic Overview

arXiv 2101.09957 ↗First posted 25 Jan 2021cs.LG cs.AI cs.NEcited by 13

Not on the map. Why?

Abstract (arXiv)

Activation functions shape the outputs of artificial neurons and, therefore, are integral parts of neural networks in general and deep learning in particular. Some activation functions, such as logistic and relu, have been used for many decades. But with deep learning becoming a mainstream research topic, new activation functions have mushroomed, leading to confusion in both theory and practice. This paper provides an analytic yet up-to-date overview of popular activation functions and their properties, which makes it a timely resource for anyone who studies or applies neural networks.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 17 of its 38 references that are in the map. First posted on arXiv 25 Jan 2021.

Builds on

17 of 38 references.

References · 17

  1. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  2. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,070
  3. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 950
  4. 2014Deep Learning in Neural Networks: An Overviewcited by 770
  5. 2017Self-Normalizing Neural Networkscited by 598
  6. 2014Convex Optimization: Algorithms and Complexitycited by 575
  7. 2015Empirical Evaluation of Rectified Activations in Convolutional Networkcited by 519
  8. 2015Norm-Based Capacity Control in Neural Networkscited by 299
  9. 2013Maxout Networkscited by 187
  10. 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 184
  11. 2019Mish: A Self Regularized Non-Monotonic Activation Functioncited by 141
  12. 2016Parametric Exponential Linear Unit for Deep Convolutional Neural Networkscited by 32
5 more
  1. 2020Statistical Guarantees for Regularized Neural Networkscited by 25
  2. 2018The Quest for the Golden Activation Functioncited by 14
  3. 2020Risk Bounds for Robust Deep Learningcited by 9
  4. 2020Layer Sparsity in Neural Networkscited by 6
  5. 2019A lattice-based approach to the expressivity of deep ReLU neural networkscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 3; 2022: 1; 2023: 2; 2024: 4; 2025: 0; 2026: 3.

13 papers.

Sort:

By area:

  1. 2021Regularization and Reparameterization Avoid Vanishing Gradients in Sigmoid-Type Networkscited by 5
  2. 2024The Definitive Guide to Policy Gradients in Deep Reinforcement Learning: Theory, Algorithms and Implementationscited by 5
  3. 2023Home Electricity Data Generator (HEDGE): An open-access tool for the generation of electric vehicle, residential demand, and PV generation profilescited by 3
  4. 2026Fermi-Dirac machines as quantizations of neuronscited by 3
  5. 2021DeepMoM: Robust Deep Learning With Median-of-Meanscited by 2
  6. 2022Energy reconstruction for large liquid scintillator detectors with machine learning techniques: aggregated features approachcited by 2
  7. 2024Interpretable machine learning approach for electron antineutrino selection in a large liquid scintillator detectorcited by 1
  8. 2021Targeted Deep Learning: Framework, Methods, and Applicationscited by 0
  9. 2023The DeepCAR Method: Forecasting Time-Series Data That Have Change Pointscited by 0
  10. 2024Nonlinearity Enhanced Adaptive Activation Functionscited by 0
  11. 2024DSReLU: A Novel Dynamic Slope Function for Superior Model Trainingcited by 0
  12. 2026IGLU: The Integrated Gaussian Linear Unit Activation Functioncited by 0
1 more
  1. 2026The Cost of Relaxation: Evaluating the Error in Convex Neural Network Verificationcited by 0