# PowerLinear Activation Functions with application to the first layer of CNNs

arXiv: https://arxiv.org/abs/2108.09256 · HTML: https://mlsanity.com/paper/2108.09256/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 12 references that are in the map.
- First posted on arXiv 20 Aug 2021.

## Builds on (most cited first, 7 of 7)

- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Network In Network](https://mlsanity.com/paper/1312.4400/) (2013)
- [Learning Activation Functions to Improve Deep Neural Networks](https://mlsanity.com/paper/1412.6830/) (2014)
- [Kervolutional Neural Networks](https://mlsanity.com/paper/1904.03955/) (2019)
- [Deep SimNets](https://mlsanity.com/paper/1506.03059/) (2015)
- [Generalizing the Convolution Operator in Convolutional Neural Networks](https://mlsanity.com/paper/1707.09864/) (2017)
- [Ensemble of Convolutional Neural Networks Trained with Different Activation Functions](https://mlsanity.com/paper/1905.02473/) (2019)

## Abstract (arXiv)

Convolutional neural networks (CNNs) have become the state-of-the-art tool for dealing with unsolved problems in computer vision and image processing. Since the convolution operator is a linear operator, several generalizations have been proposed to improve the performance of CNNs. One way to increase the capability of the convolution operator is by applying activation functions on the inner product operator. In this paper, we will introduce PowerLinear activation functions, which are based on the polynomial kernel generalization of the convolution operator. EvenPowLin functions are the main branch of the PowerLinear activation functions. This class of activation functions is saturated neither in the positive input region nor in the negative one. Also, the negative inputs are activated with the same magnitude as the positive inputs. These features made the EvenPowLin activation functions able to be utilized in the first layer of CNN architectures and learn complex features of input images. Additionally, EvenPowLin activation functions are used in CNN models to classify the inversion of grayscale images as accurately as the original grayscale images, which is significantly better than commonly used activation functions.