A Survey on Activation Functions and their relation with Xavier and He Normal Initialization
Not on the map. Why?
Abstract (arXiv)
In artificial neural network, the activation function and the weight initialization method play important roles in training and performance of a neural network. The question arises is what properties of a function are important/necessary for being a well-performing activation function. Also, the most widely used weight initialization methods - Xavier and He normal initialization have fundamental connection with activation function. This survey discusses the important/necessary properties of activation function and the most widely used activation functions (sigmoid, tanh, ReLU, LReLU and PReLU). This survey also explores the relationship between these activation functions and the two weight initialization methods - Xavier and He normal initialization.
Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). First posted on arXiv 18 Mar 2020.
Builds on
No reference list yet.
Built on by
Citing papers per year: 2021: 2; 2022: 3; 2023: 1; 2024: 0; 2025: 5; 2026: 2.
13 papers.
Sort:
- 2021Quadratic Residual Networks: A New Class of Neural Networks for Solving Forward and Inverse Problems in Physics Involving PDEscited by 42
- 2021Spatial Graph Convolutional Neural Network via Structured Subdomain Adaptation and Domain Adversarial Learning for Bearing Fault Diagnosiscited by 6
- 2022A survey on recently proposed activation functions for Deep Learningcited by 5
- 2025Motion Control in Multi-Rotor Aerial Robots Using Deep Reinforcement Learningcited by 2
- 2026Adaptive Hard-Soft Physics-Informed Neural Networks for Robust Boundary-Constrained PDE Solvingcited by 1
- 2022Comparative Analysis of State-of-the-Art Deep Learning Models for Detecting COVID-19 Lung Infection from Chest X-Ray Imagescited by 0
- 2022Evaluating CNN with Oscillatory Activation Functioncited by 0
- 2023Extending Neural Network Verification to a Larger Family of Piece-wise Linear Activation Functionscited by 0
- 2025A Study of Hybrid and Evolutionary Metaheuristics for Single Hidden Layer Feedforward Neural Network Architecturecited by 0
- 2025Exploring the Feasibility of Deep Learning Techniques for Accurate Gender Classification from Eye Imagescited by 0
- 2025Physics-Embedded Neural ODEs for Sim2Real Edge Digital Twins of Hybrid Power Electronics Systemscited by 0
- 2025Developing Training Procedures for Piecewise-linear Spline Activation Functions in Neural Networkscited by 0
- 2026Towards Investigating Residual Hearing Loss: Quantification of Fibrosis in a Novel Cochlear OCT Dataset21 Aug 2026
- 2026Adaptive Hard-Soft Physics-Informed Neural Networks for Robust Boundary-Constrained PDE Solving22 Jun 2026
- 2025Developing Training Procedures for Piecewise-linear Spline Activation Functions in Neural Networks17 Sep 2025
- 2025Physics-Embedded Neural ODEs for Sim2Real Edge Digital Twins of Hybrid Power Electronics Systems4 Aug 2025
- 2025Exploring the Feasibility of Deep Learning Techniques for Accurate Gender Classification from Eye Images31 Jul 2025
- 2025A Study of Hybrid and Evolutionary Metaheuristics for Single Hidden Layer Feedforward Neural Network Architecture17 Jun 2025
- 2025Motion Control in Multi-Rotor Aerial Robots Using Deep Reinforcement Learning9 Feb 2025
- 2023Extending Neural Network Verification to a Larger Family of Piece-wise Linear Activation Functions16 Nov 2023
- 2022Evaluating CNN with Oscillatory Activation Function13 Nov 2022
- 2022Comparative Analysis of State-of-the-Art Deep Learning Models for Detecting COVID-19 Lung Infection from Chest X-Ray Images1 Jul 2022
- 2022A survey on recently proposed activation functions for Deep Learning6 Apr 2022
- 2021Spatial Graph Convolutional Neural Network via Structured Subdomain Adaptation and Domain Adversarial Learning for Bearing Fault Diagnosis11 Dec 2021
- 2021Quadratic Residual Networks: A New Class of Neural Networks for Solving Forward and Inverse Problems in Physics Involving PDEscited by 42
- 2021Spatial Graph Convolutional Neural Network via Structured Subdomain Adaptation and Domain Adversarial Learning for Bearing Fault Diagnosiscited by 6
- 2022A survey on recently proposed activation functions for Deep Learningcited by 5
- 2025Motion Control in Multi-Rotor Aerial Robots Using Deep Reinforcement Learningcited by 2
- 2026Adaptive Hard-Soft Physics-Informed Neural Networks for Robust Boundary-Constrained PDE Solvingcited by 1
- 2022Comparative Analysis of State-of-the-Art Deep Learning Models for Detecting COVID-19 Lung Infection from Chest X-Ray Imagescited by 0
- 2022Evaluating CNN with Oscillatory Activation Functioncited by 0
- 2023Extending Neural Network Verification to a Larger Family of Piece-wise Linear Activation Functionscited by 0
- 2025A Study of Hybrid and Evolutionary Metaheuristics for Single Hidden Layer Feedforward Neural Network Architecturecited by 0
- 2025Exploring the Feasibility of Deep Learning Techniques for Accurate Gender Classification from Eye Imagescited by 0
- 2025Physics-Embedded Neural ODEs for Sim2Real Edge Digital Twins of Hybrid Power Electronics Systemscited by 0
- 2025Developing Training Procedures for Piecewise-linear Spline Activation Functions in Neural Networkscited by 0
12 most cited of 13 All 13 (JSON)
- 2026Towards Investigating Residual Hearing Loss: Quantification of Fibrosis in a Novel Cochlear OCT Dataset21 Aug 2026
- 2026Adaptive Hard-Soft Physics-Informed Neural Networks for Robust Boundary-Constrained PDE Solving22 Jun 2026
- 2025Developing Training Procedures for Piecewise-linear Spline Activation Functions in Neural Networks17 Sep 2025
- 2025Physics-Embedded Neural ODEs for Sim2Real Edge Digital Twins of Hybrid Power Electronics Systems4 Aug 2025
- 2025Exploring the Feasibility of Deep Learning Techniques for Accurate Gender Classification from Eye Images31 Jul 2025
- 2025A Study of Hybrid and Evolutionary Metaheuristics for Single Hidden Layer Feedforward Neural Network Architecture17 Jun 2025
- 2025Motion Control in Multi-Rotor Aerial Robots Using Deep Reinforcement Learning9 Feb 2025
- 2023Extending Neural Network Verification to a Larger Family of Piece-wise Linear Activation Functions16 Nov 2023
- 2022Evaluating CNN with Oscillatory Activation Function13 Nov 2022
- 2022Comparative Analysis of State-of-the-Art Deep Learning Models for Detecting COVID-19 Lung Infection from Chest X-Ray Images1 Jul 2022
- 2022A survey on recently proposed activation functions for Deep Learning6 Apr 2022
- 2021Spatial Graph Convolutional Neural Network via Structured Subdomain Adaptation and Domain Adversarial Learning for Bearing Fault Diagnosis11 Dec 2021
12 newest of 13 All 13 (JSON)