mlsanityData to 7 Oct 2026

Activation Functions: Comparison of trends in Practice and Research for Deep Learning

arXiv 1811.03378 ↗First posted 8 Nov 2018cs.LG cs.CVcited by 130

Not on the map. Why?

Abstract (arXiv)

Deep neural networks have been successfully used in diverse emerging domains to solve real world complex problems with may more deep learning(DL) architectures, being developed to date. To achieve these state-of-the-art performances, the DL architectures use activation functions (AFs), to perform diverse computations between the hidden layers and the output layers of any given DL architecture. This paper presents a survey on the existing AFs used in deep learning applications and highlights the recent trends in the use of the activation functions for deep learning applications. The novelty of this paper is that it compiles majority of the AFs used in DL and outlines the current trends in the applications and usage of these functions in practical deep learning deployments against the state-of-the-art research results. This compilation will aid in making effective decisions in the choice of the most suitable and appropriate activation function for any given application, ready for deployment. This paper is timely because most research papers on AF highlights similar works and results while this paper will be the first, to compile the trends in AF applications in practice against the research results from literature, found in deep learning research to date.

Key facts. Cited by 130 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 Nov 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 33; 2022: 32; 2023: 24; 2024: 19; 2025: 12; 2026: 10.

130 papers.

Sort:

By area:

  1. 2021E(n) Equivariant Normalizing Flowscited by 165
  2. 2021Activation Functions in Deep Learning: A Comprehensive Survey and Benchmarkcited by 153
  3. 2021ReduNet: A White-box Deep Network from the Principle of Maximizing Rate Reductioncited by 84
  4. 2023DiffusionDepth: Diffusion Denoising Approach for Monocular Depth Estimationcited by 81
  5. 2022Evaluating the Susceptibility of Pre-Trained Language Models via Handcrafted Adversarial Examplescited by 65
  6. 2021Model Complexity of Deep Learning: A Surveycited by 64
  7. 2021DPFM: Deep Partial Functional Mapscited by 58
  8. 2021A Review on Deep Learning in UAV Remote Sensingcited by 32
  9. 2022How important are activation functions in regression and classification? A survey, performance comparison, and future directionscited by 29
  10. 2023RMP-Loss: Regularizing Membrane Potential Distribution for Spiking Neural Networkscited by 27
  11. 2021Generative Modeling of Turbulencecited by 26
  12. 2022PLIKS: A Pseudo-Linear Inverse Kinematic Solver for 3D Human Body Estimationcited by 18
38 more
  1. 2021Medical Image Segmentation with 3D Convolutional Neural Networks: A Surveycited by 16
  2. 2021Minimal Multi-Layer Modifications of Deep Neural Networkscited by 14
  3. 2021Privacy and Trust Redefined in Federated Machine Learningcited by 13
  4. 2022Universal Spam Detection using Transfer Learning of BERT Modelcited by 13
  5. 2021FitAct: Error Resilient Deep Neural Networks via Fine-Grained Post-Trainable Activation Functionscited by 12
  6. 2022Concept-Based Techniques for "Musicologist-friendly" Explanations in a Deep Music Classifiercited by 12
  7. 2022Federated Learning and Meta Learning: Approaches, Applications, and Directionscited by 12
  8. 2024Motion meets Attention: Video Motion Promptscited by 12
  9. 2021Growing Cosine Unit: A Novel Oscillatory Activation Function That Can Speedup Training and Reduce Parameters in Convolutional Neural Networkscited by 11
  10. 2022Convergence of a robust deep FBSDE method for stochastic controlcited by 11
  11. 2024SIGMA: Selective Gated Mamba for Sequential Recommendationcited by 10
  12. 2021Automatic Fault Detection for Deep Learning Programs Using Graph Transformationscited by 9
  13. 2023SiCP: Simultaneous Individual and Cooperative Perception for 3D Object Detection in Connected and Automated Vehiclescited by 8
  14. 2023Deep Generative Models, Synthetic Tabular Data, and Differential Privacy: An Overview and Synthesiscited by 7
  15. 2021Non-intrusive Surrogate Modeling for Parametrized Time-dependent PDEs using Convolutional Autoencoderscited by 6
  16. 2021Gradient Forward-Propagation for Large-Scale Temporal Video Modellingcited by 5
  17. 2021Software Vulnerability Detection via Deep Learning over Disaggregated Code Graph Representationcited by 5
  18. 2021AutoInit: Analytic Signal-Preserving Weight Initialization for Neural Networkscited by 5
  19. 2022textitFastSVD-ML-ROM: A Reduced-Order Modeling Framework based on Machine Learning for Real-Time Applicationscited by 5
  20. 2022Transformers with Learnable Activation Functionscited by 5
  21. 2022Medical Diagnosis with Large Scale Multimodal Transformers: Leveraging Diverse Data for More Accurate Diagnosiscited by 5
  22. 2023Efficient Activation Function Optimization through Surrogate Modelingcited by 5
  23. 2023Neuroscience inspired scientific machine learning (Part-1): Variable spiking neuron for regressioncited by 5
  24. 2022Combining Varied Learners for Binary Classification using Stacked Generalizationcited by 4
  25. 2022Neural Networks with A La Carte Selection of Activation Functionscited by 4
  26. 2022Trusted Multi-Scale Classification Framework for Whole Slide Imagecited by 4
  27. 2022SignReLU neural network and its approximation abilitycited by 4
  28. 2024HEAD: A Bandwidth-Efficient Cooperative Perception Approach for Heterogeneous Connected and Autonomous Vehiclescited by 4
  29. 2024Quadratic Gating Mixture of Experts: Statistical Insights into Self-Attentioncited by 4
  30. 2022Predicting the Geoeffectiveness of CMEs Using Machine Learningcited by 3
  31. 2023Amplifying Sine Unit: An Oscillatory Activation Function for Deep Neural Networks to Recover Nonlinear Oscillations Efficientlycited by 3
  32. 2023NeuroCodeBench: a plain C neural network benchmark for software verificationcited by 3
  33. 2023Optimal Nonlinearities Improve Generalization Performance of Random Featurescited by 3
  34. 2021Evading Malware Classifiers via Monte Carlo Mutant Feature Discoverycited by 2
  35. 2021Multiagent Model-based Credit Assignment for Continuous Controlcited by 2
  36. 2022Real World Large Scale Recommendation Systems Reproducibility and Smooth Activationscited by 2
  37. 2022Residual-Concatenate Neural Network with Deep Regularization Layers for Binary Classificationcited by 2
  38. 2022Forecasting Algorithms for Causal Inference with Panel Datacited by 2

50 most cited of 130 All 130 (JSON)