# Comparison of non-linear activation functions for deep neural networks on MNIST classification task

arXiv: https://arxiv.org/abs/1804.02763 · HTML: https://mlsanity.com/paper/1804.02763/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Apr 2018.

## Built on by (most cited first, 19 of 19)

- [Activation Functions in Deep Learning: A Comprehensive Survey and Benchmark](https://mlsanity.com/paper/2109.14545/) (2021)
- [Beyond Periodicity: Towards a Unifying Framework for Activations in Coordinate-MLPs](https://mlsanity.com/paper/2111.15135/) (2021)
- [How important are activation functions in regression and classification? A survey, performance comparison, and future directions](https://mlsanity.com/paper/2209.02681/) (2022)
- [Sparse Distributed Memory is a Continual Learner](https://mlsanity.com/paper/2303.11934/) (2023)
- [A Survey on Machine Learning Techniques for Auto Labeling of Video, Audio, and Text Data](https://mlsanity.com/paper/2109.03784/) (2021)
- [Simple and complex spiking neurons: perspectives and analysis in a simple STDP scenario](https://mlsanity.com/paper/2207.04881/) (2022)
- [An Inexact Augmented Lagrangian Algorithm for Training Leaky ReLU Neural Network with Group Sparsity](https://mlsanity.com/paper/2205.05428/) (2022)
- [Efficient Quantum Convolutional Neural Networks for Image Classification: Overcoming Hardware Constraints](https://mlsanity.com/paper/2505.05957/) (2025)
- [Semi-Supervised Confidence-Level-based Contrastive Discrimination for Class-Imbalanced Semantic Segmentation](https://mlsanity.com/paper/2211.15066/) (2022)
- [A New Multifractal-based Deep Learning Model for Text Mining](https://mlsanity.com/paper/2111.13861/) (2021)
- [Real World Large Scale Recommendation Systems Reproducibility and Smooth Activations](https://mlsanity.com/paper/2202.06499/) (2022)
- [A Hybrid Brain-Computer Interface Using Motor Imagery and SSVEP Based on Convolutional Neural Network](https://mlsanity.com/paper/2212.05289/) (2022)
- [Single- vs. Dual-Policy Reinforcement Learning for Dynamic Bike Rebalancing](https://mlsanity.com/paper/2402.03589/) (2024)
- [Dual Policy Reinforcement Learning for Real-time Rebalancing in Bike-sharing Systems](https://mlsanity.com/paper/2406.00868/) (2024)
- [CGAN-EB: A Non-parametric Empirical Bayes Method for Crash Hotspot Identification Using Conditional Generative Adversarial Networks: A Real-world Crash Data Study](https://mlsanity.com/paper/2112.10588/) (2021)
- [Crash Data Augmentation Using Conditional Generative Adversarial Networks (CGAN) for Improving Safety Performance Functions](https://mlsanity.com/paper/2112.12263/) (2021)
- [CGAN-EB: A Non-parametric Empirical Bayes Method for Crash Hotspot Identification Using Conditional Generative Adversarial Networks: A Simulated Crash Data Study](https://mlsanity.com/paper/2112.06925/) (2021)
- [Learning Dynamic Representations via An Optimally-Weighted Maximum Mean Discrepancy Optimization Framework for Continual Learning](https://mlsanity.com/paper/2501.12121/) (2025)
- [FlexAct: Why Learn when you can Pick?](https://mlsanity.com/paper/2601.06441/) (2026)

## Abstract (arXiv)

Activation functions play a key role in neural networks so it becomes fundamental to understand their advantages and disadvantages in order to achieve better performances. This paper will first introduce common types of non linear activation functions that are alternative to the well known sigmoid function and then evaluate their characteristics. Moreover deeper neural networks will be analysed because they positively influence the final performances compared to shallower networks. They also strictly depend on the weight initialisation hence the effect of drawing weights from Gaussian and uniform distribution will be analysed making particular attention on how the number of incoming and outgoing connection to a node influence the whole network.