# Comparison of different convolutional neural network activation functions and methods for building ensembles

arXiv: https://arxiv.org/abs/2103.15898 · HTML: https://mlsanity.com/paper/2103.15898/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 45 references that are in the map.
- First posted on arXiv 29 Mar 2021.

## Builds on (most cited first, 17 of 17)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [Deep Speech: Scaling up end-to-end speech recognition](https://mlsanity.com/paper/1412.5567/) (2014)
- [Deep Learning Face Representation by Joint Identification-Verification](https://mlsanity.com/paper/1406.4773/) (2014)
- [Learning Activation Functions to Improve Deep Neural Networks](https://mlsanity.com/paper/1412.6830/) (2014)
- [Deep Learning with S-shaped Rectified Linear Activation Units](https://mlsanity.com/paper/1512.07030/) (2015)
- [Learning Combinations of Activation Functions](https://mlsanity.com/paper/1801.09403/) (2018)
- [PLU: The Piecewise Linear Unit Activation Function](https://mlsanity.com/paper/1809.09534/) (2018)
- [Ensemble of Convolutional Neural Networks Trained with Different Activation Functions](https://mlsanity.com/paper/1905.02473/) (2019)
- [Soft-Root-Sign Activation Function](https://mlsanity.com/paper/2003.00547/) (2020)
- [Kafnets: kernel-based non-parametric activation functions for neural networks](https://mlsanity.com/paper/1707.04035/) (2017)

## Built on by (most cited first, 2 of 2)

- [Deep ensembles in bioimage segmentation](https://mlsanity.com/paper/2112.12955/) (2021)
- [dynActivation: A Trainable Activation Family for Adaptive Nonlinearity](https://mlsanity.com/paper/2603.22154/) (2026)

## Abstract (arXiv)

Recently, much attention has been devoted to finding highly efficient and powerful activation functions for CNN layers. Because activation functions inject different nonlinearities between layers that affect performance, varying them is one method for building robust ensembles of CNNs. The objective of this study is to examine the performance of CNN ensembles made with different activation functions, including six new ones presented here: 2D Mexican ReLU, TanELU, MeLU+GaLU, Symmetric MeLU, Symmetric GaLU, and Flexible MeLU. The highest performing ensemble was built with CNNs having different activation layers that randomly replaced the standard ReLU. A comprehensive evaluation of the proposed approach was conducted across fifteen biomedical data sets representing various classification tasks. The proposed method was tested on two basic CNN architectures: Vgg16 and ResNet50. Results demonstrate the superiority in performance of this approach. The MATLAB source code for this study will be available at https://github.com/LorisNanni.