mlsanityData to 7 Oct 2026

Activation Functions in Deep Learning: A Comprehensive Survey and Benchmark

arXiv 2109.14545 ↗First posted 29 Sep 2021cs.LG cs.NEcited by 153

Not on the map. Why?

Abstract (arXiv)

Neural networks have shown tremendous growth in recent years to solve numerous problems. Various types of neural networks have been introduced to deal with different types of problems. However, the main goal of any neural network is to transform the non-linearly separable input data into more linearly separable abstract features using a hierarchy of layers. These layers are combinations of linear and nonlinear functions. The most popular and common non-linearity layers are activation functions (AFs), such as Logistic Sigmoid, Tanh, ReLU, ELU, Swish and Mish. In this paper, a comprehensive overview and survey is presented for AFs in neural networks for deep learning. Different classes of AFs such as Logistic Sigmoid and Tanh based, ReLU based, ELU based, and Learning based are covered. Several characteristics of AFs such as output range, monotonicity, and smoothness are also pointed out. A performance comparison is also performed among 18 state-of-the-art AFs with different networks on different types of data. The insights of AFs are presented to benefit the researchers for doing further research and practitioners to select among different choices. The code used for experimental comparison is released at: urlhttps://github.com/shivram1987/ActivationFunctions.

Key facts. Cited by 153 papers in the map (data through 7 Oct 2026). Builds on 79 of its 159 references that are in the map. First posted on arXiv 29 Sep 2021.

Builds on

79 of 159 references.

References · 79

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2016Densely Connected Convolutional Networkscited by 6,691
  4. 2017Squeeze-and-Excitation Networkscited by 4,763
  5. 2014Going Deeper with Convolutionscited by 4,231
  6. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  7. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  8. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  9. 2020YOLOv4: Optimal Speed and Accuracy of Object Detectioncited by 1,624
  10. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
  11. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 935
  12. 2013Speech Recognition with Deep Recurrent Neural Networkscited by 666
38 more
  1. 2017Self-Normalizing Neural Networkscited by 591
  2. 2016Error bounds for approximations with deep ReLU networkscited by 567
  3. 2015Empirical Evaluation of Rectified Activations in Convolutional Networkcited by 515
  4. 2018Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problemcited by 323
  5. 2016Understanding Deep Neural Networks with Rectified Linear Unitscited by 253
  6. 2020Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networkscited by 227
  7. 2013Maxout Networkscited by 186
  8. 2019Mish: A Self Regularized Non-Monotonic Activation Functioncited by 141
  9. 2017Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practicecited by 132
  10. 2018Activation Functions: Comparison of trends in Practice and Research for Deep Learningcited by 130
  11. 2020A survey on modern trainable activation functionscited by 89
  12. 2016Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Unitscited by 86
  13. 2020Rational neural networkscited by 75
  14. 2017Continuously Differentiable Exponential Linear Unitscited by 73
  15. 2020A Decade Survey of Content Based Image Retrieval using Deep Learningcited by 72
  16. 2019Pad'e Activation Units: End-to-end Learning of Flexible Activation Functions in Deep Networkscited by 65
  17. 2014Learning Activation Functions to Improve Deep Neural Networkscited by 62
  18. 2020Dynamic ReLUcited by 55
  19. 2016Noisy Activation Functionscited by 53
  20. 2017Neural networks and rational functionscited by 40
  21. 2019Learning Activation Functions: A new paradigm for understanding Neural Networkscited by 33
  22. 2016Parametric Exponential Linear Unit for Deep Convolutional Neural Networkscited by 32
  23. 2019Time/Accuracy Tradeoffs for Learning a ReLU with respect to Gaussian Marginalscited by 32
  24. 2019diffGrad: An Optimization Method for Convolutional Neural Networkscited by 29
  25. 2015Deep Learning with S-shaped Rectified Linear Activation Unitscited by 26
  26. 2020Review and Comparison of Commonly Used Activation Functions for Deep Neural Networkscited by 21
  27. 2018Learning Combinations of Activation Functionscited by 20
  28. 2018Complex-valued Neural Networks with Non-parametric Activation Functionscited by 19
  29. 2018Comparison of non-linear activation functions for deep neural networks on MNIST classification taskcited by 19
  30. 2019Impact of Fully Connected Layers on Performance of Convolutional Neural Networks for Image Classificationcited by 19
  31. 2018A comparison of deep networks with ReLU activation function and linear spline-type methodscited by 18
  32. 2018The Quest for the Golden Activation Functioncited by 14
  33. 2016Learning activation functions from data using cubic spline interpolationcited by 13
  34. 2018Adaptive Blending Units: Trainable Activation Functions for Deep Neural Networkscited by 13
  35. 2019ProbAct: A Probabilistic Activation Function for Deep Neural Networkscited by 12
  36. 2019PowerNet: Efficient Representations of Polynomials and Smooth Functions by Deep Neural Networks with Rectified Power Unitscited by 12
  37. 2016Revise Saturated Activation Functionscited by 11
  38. 2018PLU: The Piecewise Linear Unit Activation Functioncited by 11

50 most cited of 79 All 79 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 5; 2023: 25; 2024: 51; 2025: 48; 2026: 23.

153 papers.

Sort:

By area:

  1. 2024Chebyshev Polynomial-Based Kolmogorov-Arnold Networks: An Efficient Architecture for Nonlinear Function Approximationcited by 115
  2. 2024Neural Redshift: Random Networks are not Random Functionscited by 43
  3. 2023When Deep Learning Meets Polyhedral Theory: A Surveycited by 39
  4. 2025Systemizing Multiplicity: The Curious Case of Arbitrariness in Machine Learningcited by 17
  5. 2023GELU Activation Function in Deep Learning: A Comprehensive Mathematical Analysis and Performancecited by 16
  6. 2024Flex-PE: Flexible and SIMD Multi-Precision Processing Element for AI Workloadscited by 15
  7. 2024HOSC: A Periodic Activation Function for Preserving Sharp Features in Implicit Neural Representationscited by 14
  8. 2025Do We Always Need the Simplicity Bias? Looking for Optimal Inductive Biases in the Wildcited by 13
  9. 2023Multigrid-Augmented Deep Learning Preconditioners for the Helmholtz Equation using Compact Implicit Layerscited by 11
  10. 2023DemaFormer: Damped Exponential Moving Average Transformer with Energy-Based Modeling for Temporal Language Groundingcited by 11
  11. 2024Noise Diffusion for Enhancing Semantic Faithfulness in Text-to-Image Synthesiscited by 9
  12. 2023A physics-informed and attention-based graph learning approach for regional electric vehicle charging demand predictioncited by 8
38 more
  1. 2024M3BAT: Unsupervised Domain Adaptation for Multimodal Mobile Sensing with Multi-Branch Adversarial Trainingcited by 8
  2. 2023Improved weight initialization for deep and narrow feedforward neural networkcited by 7
  3. 2024A Survey on Failure Analysis and Fault Injection in AI Systemscited by 7
  4. 2024Deep Learning in Medical Image Classification from MRI-based Brain Tumor Imagescited by 7
  5. 2024Towards Symbolic XAI -- Explanation Through Human Understandable Logical Relationships Between Featurescited by 7
  6. 2025Advancing Constrained Monotonic Neural Networks: Achieving Universal Approximation Beyond Bounded Activationscited by 7
  7. 2023HRTF upsampling with a generative adversarial network using a gnomonic equiangular projectioncited by 6
  8. 2024Trainable Highly-expressive Activation Functionscited by 6
  9. 2024KPCA-CAM: Visual Explainability of Deep Computer Vision Models using Kernel PCAcited by 6
  10. 2024User-Aware Multilingual Abusive Content Detection in Social Mediacited by 6
  11. 2022Transformers with Learnable Activation Functionscited by 5
  12. 2023Initial Guessing Bias: How Untrained Networks Favor Some Classescited by 5
  13. 2024Kolmogorov-Arnold Networks in Low-Data Regimes: A Comparative Study with Multilayer Perceptronscited by 5
  14. 2026All elementary functions from a single binary operatorcited by 5
  15. 2022SignReLU neural network and its approximation abilitycited by 4
  16. 2023Generalizable End-to-End Deep Learning Frameworks for Real-Time Attitude Estimation Using 6DoF Inertial Measurement Unitscited by 4
  17. 2023Universal Scaling Laws of Absorbing Phase Transitions in Artificial Deep Neural Networkscited by 4
  18. 2023The Geometric Structure of Fully-Connected ReLU Layerscited by 4
  19. 2024Source-Free Cross-Modal Knowledge Transfer by Unleashing the Potential of Task-Irrelevant Datacited by 4
  20. 2024Speech Swin-Transformer: Exploring a Hierarchical Transformer with Shifted Windows for Speech Emotion Recognitioncited by 4
  21. 2024Bridging scales in multiscale bubble growth dynamics with correlated fluctuations using neural operator learningcited by 4
  22. 2024The Cost of Arbitrariness for Individuals: Examining the Legal and Technical Challenges of Model Multiplicitycited by 4
  23. 2024Cauchy activation function and XNetcited by 4
  24. 2024Nonlinear thermodynamic computing out of equilibriumcited by 4
  25. 2025Proof Minimization in Neural Network Verificationcited by 4
  26. 2023ErfReLU: Adaptive Activation Function for Deep Neural Networkcited by 3
  27. 2024MelNet: A Real-Time Deep Learning Algorithm for Object Detectioncited by 3
  28. 2024Adaptive Activation Functions for Predictive Modeling with Sparse Experimental Datacited by 3
  29. 2024Image Classification with Rotation-Invariant Variational Quantum Circuitscited by 3
  30. 2024What comes after transformers? -- A selective survey connecting ideas in deep learningcited by 3
  31. 2024Hard-Label Cryptanalytic Extraction of Neural Network Modelscited by 3
  32. 2024Exploiting Memory-aware Q-distribution Prediction for Nuclear Fusion via Modern Hopfield Networkcited by 3
  33. 2024Quantum Pointwise Convolution: A Flexible and Scalable Approach for Neural Network Enhancementcited by 3
  34. 2025Sparse Brains are Also Adaptive Brains: Cognitive-Load-Aware Dynamic Activation for LLMscited by 3
  35. 2025Beyond ReLU: How Activations Affect Neural Kernels and Random Wide Networkscited by 3
  36. 2026Mining Generalizable Activation Functionscited by 3
  37. 2021A New Multifractal-based Deep Learning Model for Text Miningcited by 2
  38. 2023Empirical study of the modulus as activation function in computer vision applicationscited by 2

50 most cited of 153 All 153 (JSON)