mlsanityData to 7 Oct 2026

How important are activation functions in regression and classification? A survey, performance comparison, and future directions

arXiv 2209.02681 ↗First posted 6 Sep 2022cs.LGcited by 29

Not on the map. Why?

Abstract (arXiv)

Inspired by biological neurons, the activation functions play an essential part in the learning process of any artificial neural network commonly used in many real-world problems. Various activation functions have been proposed in the literature for classification as well as regression tasks. In this work, we survey the activation functions that have been employed in the past as well as the current state-of-the-art. In particular, we present various developments in activation functions over the years and the advantages as well as disadvantages or limitations of these activation functions. We also discuss classical (fixed) activation functions, including rectifier units, and adaptive activation functions. In addition to discussing the taxonomy of activation functions based on characterization, a taxonomy of activation functions based on applications is presented. To this end, the systematic comparison of various fixed and adaptive activation functions is performed for classification data sets such as the MNIST, CIFAR-10, and CIFAR- 100. In recent years, a physics-informed machine learning framework has emerged for solving problems related to scientific computations. For this purpose, we also discuss various requirements for activation functions that have been used in the physics-informed machine learning framework. Furthermore, various comparisons are made among different fixed and adaptive activation functions using various machine learning libraries such as TensorFlow, Pytorch, and JAX.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 77 of its 222 references that are in the map. First posted on arXiv 6 Sep 2022.

Builds on

77 of 222 references.

References · 77

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  5. 2018Neural Ordinary Differential Equationscited by 3,733
  6. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  7. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  8. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  9. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
  10. 2015Automatic differentiation in machine learning: a surveycited by 846
  11. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
  12. 2017Self-Normalizing Neural Networkscited by 591
38 more
  1. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 571
  2. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 557
  3. 2015Empirical Evaluation of Rectified Activations in Convolutional Networkcited by 515
  4. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 456
  5. 2015Unitary Evolution Recurrent Neural Networkscited by 291
  6. 2017Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weightscited by 211
  7. 2020ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learningcited by 201
  8. 2013Maxout Networkscited by 186
  9. 2016Theory-guided Data Science: A New Paradigm for Scientific Discovery from Datacited by 169
  10. 2019Mish: A Self Regularized Non-Monotonic Activation Functioncited by 141
  11. 2018Activation Functions: Comparison of trends in Practice and Research for Deep Learningcited by 130
  12. 2017Deep Learning with Low Precision by Half-wave Gaussian Quantizationcited by 115
  13. 2021Parallel Physics-Informed Neural Networks via Domain Decompositioncited by 112
  14. 2020A survey on modern trainable activation functionscited by 89
  15. 2016Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Unitscited by 86
  16. 2022Physics-informed neural networks for inverse problems in supersonic flowscited by 75
  17. 2018Deep Learning of Vortex Induced Vibrationscited by 74
  18. 2017Continuously Differentiable Exponential Linear Unitscited by 73
  19. 2015Neural Network with Unbounded Activation Functions is Universal Approximatorcited by 63
  20. 2022Error estimates for physics informed neural networks approximating the Navier-Stokes equationscited by 63
  21. 2014Learning Activation Functions to Improve Deep Neural Networkscited by 62
  22. 2021When Do Extended Physics-Informed Neural Networks (XPINNs) Improve Generalization?cited by 57
  23. 2016Noisy Activation Functionscited by 53
  24. 2017WRPN: Wide Reduced-Precision Networkscited by 49
  25. 2021Deep Kronecker neural networks: A general framework for neural networks with adaptive activation functionscited by 49
  26. 2019Learning Activation Functions: A new paradigm for understanding Neural Networkscited by 33
  27. 2016Parametric Exponential Linear Unit for Deep Convolutional Neural Networkscited by 32
  28. 2017Better than Real: Complex-valued Neural Nets for MRI Fingerprintingcited by 30
  29. 2015Deep Learning with S-shaped Rectified Linear Activation Unitscited by 26
  30. 2016On Complex Valued Convolutional Neural Networkscited by 26
  31. 2020Discovering Parametric Activation Functionscited by 21
  32. 2020Review and Comparison of Commonly Used Activation Functions for Deep Neural Networkscited by 21
  33. 2018Complex-valued Neural Networks with Non-parametric Activation Functionscited by 19
  34. 2018Comparison of non-linear activation functions for deep neural networks on MNIST classification taskcited by 19
  35. 2018Defend Deep Neural Networks Against Adversarial Examples via Fixed and Dynamic Quantized Activation Functionscited by 17
  36. 2018The Quest for the Golden Activation Functioncited by 14
  37. 2018Adaptive Blending Units: Trainable Activation Functions for Deep Neural Networkscited by 13
  38. 2016Deep Residual Networks with Exponential Linear Unitcited by 12

50 most cited of 77 All 77 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 10; 2024: 7; 2025: 6; 2026: 4.

29 papers.

Sort:

  1. 2024fKAN: Fractional Kolmogorov-Arnold Networks with trainable Jacobi basis functionscited by 60
  2. 2023A unified scalable framework for causal sweeping strategies for Physics-Informed Neural Networks (PINNs) and their temporal decompositionscited by 48
  3. 2023Learning stiff chemical kinetics using extended deep neural operatorscited by 30
  4. 2024RiemannONets: Interpretable Neural Operators for Riemann Problemscited by 28
  5. 2022Modular machine learning-based elastoplasticity: generalization in the context of limited datacited by 16
  6. 2023Mixed formulation of physics-informed neural networks for thermo-mechanically coupled systems and heterogeneous domainscited by 14
  7. 2025Do We Always Need the Simplicity Bias? Looking for Optimal Inductive Biases in the Wildcited by 13
  8. 2025Challenges and Advancements in Modeling Shock Fronts with Physics-Informed Neural Networks: A Review and Benchmarking Studycited by 11
  9. 2025Anant-Net: Breaking the Curse of Dimensionality with Scalable and Interpretable Neural Surrogate for High-Dimensional PDEscited by 11
  10. 2022Physics-informed Neural Networks with Periodic Activation Functions for Solute Transport in Heterogeneous Porous Mediacited by 8
  11. 2023Learning solution of nonlinear constitutive material models using physics-informed neural networks: COMM-PINNcited by 6
  12. 2023Physics-informed neural networks for predicting gas flow dynamics and unknown parameters in diesel enginescited by 6
17 more
  1. 2025Complex Physics-Informed Neural Networkcited by 6
  2. 2024Kolmogorov-Arnold Networks in Low-Data Regimes: A Comparative Study with Multilayer Perceptronscited by 5
  3. 2025BubbleOKAN: A Physics-Informed Interpretable Neural Operator for High-Frequency Bubble Dynamicscited by 5
  4. 2026FEKAN: Feature-Enriched Kolmogorov-Arnold Networkscited by 5
  5. 2023Self-Supervised Learning for Data Scarcity in a Fatigue Damage Prognostic Problemcited by 3
  6. 2023iPINNs: Incremental learning for Physics-informed neural networkscited by 3
  7. 2024Adaptive Activation Functions for Predictive Modeling with Sparse Experimental Datacited by 3
  8. 2024Probabilistic Neural Networks (PNNs) for Modeling Aleatoric Uncertainty in Scientific Machine Learningcited by 2
  9. 2026Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Taskscited by 2
  10. 2026Tackling Failure Modes of PINNs and PIKANs Using Conflict-Free Gradientscited by 2
  11. 2023Deep smoothness WENO scheme for two-dimensional hyperbolic conservation laws: A deep learning approach for learning smoothness indicatorscited by 1
  12. 2023The limitation of neural nets for approximation and optimizationcited by 1
  13. 2024Kolmogorov n-Widths for Multitask Physics-Informed Machine Learning (PIML) Methods: Towards Robust Metricscited by 1
  14. 2026In Transformer We Trust? A Perspective on Transformer Architecture Failure Modescited by 1
  15. 2023Data-driven localized waves and parameter discovery in the massive Thirring model via extended physics-informed neural networks with interface zonescited by 0
  16. 2024Nonlinearity Enhanced Adaptive Activation Functionscited by 0
  17. 2025Robust Deep Network Learning of Nonlinear Regression Tasks by Parametric Leaky Exponential Linear Units (LELUs) and a Diffusion Metriccited by 0