# A Survey on Activation Functions and their relation with Xavier and He Normal Initialization

arXiv: https://arxiv.org/abs/2004.06632 · HTML: https://mlsanity.com/paper/2004.06632/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Mar 2020.

## Built on by (most cited first, 13 of 13)

- [Quadratic Residual Networks: A New Class of Neural Networks for Solving Forward and Inverse Problems in Physics Involving PDEs](https://mlsanity.com/paper/2101.08366/) (2021)
- [Spatial Graph Convolutional Neural Network via Structured Subdomain Adaptation and Domain Adversarial Learning for Bearing Fault Diagnosis](https://mlsanity.com/paper/2112.06033/) (2021)
- [A survey on recently proposed activation functions for Deep Learning](https://mlsanity.com/paper/2204.02921/) (2022)
- [Motion Control in Multi-Rotor Aerial Robots Using Deep Reinforcement Learning](https://mlsanity.com/paper/2502.05996/) (2025)
- [Adaptive Hard-Soft Physics-Informed Neural Networks for Robust Boundary-Constrained PDE Solving](https://mlsanity.com/paper/2606.23359/) (2026)
- [Comparative Analysis of State-of-the-Art Deep Learning Models for Detecting COVID-19 Lung Infection from Chest X-Ray Images](https://mlsanity.com/paper/2208.01637/) (2022)
- [Evaluating CNN with Oscillatory Activation Function](https://mlsanity.com/paper/2211.06878/) (2022)
- [Extending Neural Network Verification to a Larger Family of Piece-wise Linear Activation Functions](https://mlsanity.com/paper/2311.10780/) (2023)
- [A Study of Hybrid and Evolutionary Metaheuristics for Single Hidden Layer Feedforward Neural Network Architecture](https://mlsanity.com/paper/2506.15737/) (2025)
- [Exploring the Feasibility of Deep Learning Techniques for Accurate Gender Classification from Eye Images](https://mlsanity.com/paper/2508.00135/) (2025)
- [Physics-Embedded Neural ODEs for Sim2Real Edge Digital Twins of Hybrid Power Electronics Systems](https://mlsanity.com/paper/2508.02887/) (2025)
- [Developing Training Procedures for Piecewise-linear Spline Activation Functions in Neural Networks](https://mlsanity.com/paper/2509.18161/) (2025)
- [Towards Investigating Residual Hearing Loss: Quantification of Fibrosis in a Novel Cochlear OCT Dataset](https://mlsanity.com/paper/2608.21189/) (2026)

## Abstract (arXiv)

In artificial neural network, the activation function and the weight initialization method play important roles in training and performance of a neural network. The question arises is what properties of a function are important/necessary for being a well-performing activation function. Also, the most widely used weight initialization methods - Xavier and He normal initialization have fundamental connection with activation function. This survey discusses the important/necessary properties of activation function and the most widely used activation functions (sigmoid, tanh, ReLU, LReLU and PReLU). This survey also explores the relationship between these activation functions and the two weight initialization methods - Xavier and He normal initialization.