# An Analysis of State-of-the-art Activation Functions For Supervised Deep Neural Network

arXiv: https://arxiv.org/abs/2104.02523 · HTML: https://mlsanity.com/paper/2104.02523/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 19 references that are in the map.
- First posted on arXiv 5 Apr 2021.

## Builds on (most cited first, 10 of 10)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [Self-Normalizing Neural Networks](https://mlsanity.com/paper/1706.02515/) (2017)
- [Do Deep Nets Really Need to be Deep?](https://mlsanity.com/paper/1312.6184/) (2013)
- [A multi-device dataset for urban acoustic scene classification](https://mlsanity.com/paper/1807.09840/) (2018)
- [Robust Acoustic Scene Classification using a Multi-Spectrogram Encoder-Decoder Framework](https://mlsanity.com/paper/2002.04502/) (2020)
- [Improving Deep Learning by Inverse Square Root Linear Units (ISRLUs)](https://mlsanity.com/paper/1710.09967/) (2017)
- [Sound Context Classification Basing on Join Learning Model and Multi-Spectrogram Features](https://mlsanity.com/paper/2005.12779/) (2020)

## Built on by (most cited first, 4 of 4)

- [AGTCNet: A Graph-Temporal Approach for Principled Motor Imagery EEG Classification](https://mlsanity.com/paper/2506.21338/) (2025)
- [Targeting the partition function of chemically disordered materials with a generative approach based on inverse variational autoencoders](https://mlsanity.com/paper/2408.14928/) (2024)
- [A New Paradigm for Device-free Indoor Localization: Deep Learning with Error Vector Spectrum in Wi-Fi Systems](https://mlsanity.com/paper/2304.06490/) (2023)
- [Spatially Optimized Compact Deep Metric Learning Model for Similarity Search](https://mlsanity.com/paper/2404.06593/) (2024)

## Abstract (arXiv)

This paper provides an analysis of state-of-the-art activation functions with respect to supervised classification of deep neural network. These activation functions comprise of Rectified Linear Units (ReLU), Exponential Linear Unit (ELU), Scaled Exponential Linear Unit (SELU), Gaussian Error Linear Unit (GELU), and the Inverse Square Root Linear Unit (ISRLU). To evaluate, experiments over two deep learning network architectures integrating these activation functions are conducted. The first model, basing on Multilayer Perceptron (MLP), is evaluated with MNIST dataset to perform these activation functions. Meanwhile, the second model, likely VGGish-based architecture, is applied for Acoustic Scene Classification (ASC) Task 1A in DCASE 2018 challenge, thus evaluate whether these activation functions work well in different datasets as well as different network architectures.