# Empirical analysis of non-linear activation functions for Deep Neural Networks in classification tasks

arXiv: https://arxiv.org/abs/1710.11272 · HTML: https://mlsanity.com/paper/1710.11272/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Oct 2017.

## Built on by (most cited first, 3 of 3)

- [Activation Functions in Deep Learning: A Comprehensive Survey and Benchmark](https://mlsanity.com/paper/2109.14545/) (2021)
- [Graph Reinforcement Learning-based CNN Inference Offloading in Dynamic Edge Computing](https://mlsanity.com/paper/2210.13464/) (2022)
- [Dynamic Semantic Compression for CNN Inference in Multi-access Edge Computing: A Graph Reinforcement Learning-based Autoencoder](https://mlsanity.com/paper/2401.12167/) (2024)

## Abstract (arXiv)

We provide an overview of several non-linear activation functions in a neural network architecture that have proven successful in many machine learning applications. We conduct an empirical analysis on the effectiveness of using these function on the MNIST classification task, with the aim of clarifying which functions produce the best results overall. Based on this first set of results, we examine the effects of building deeper architectures with an increasing number of hidden layers. We also survey the impact of using, on the same task, different initialisation schemes for the weights of our neural network. Using these sets of experiments as a base, we conclude by providing a optimal neural network architecture that yields impressive results in accuracy on the MNIST classification task.