# Parametric Exponential Linear Unit for Deep Convolutional Neural Networks

arXiv: https://arxiv.org/abs/1605.09332 · HTML: https://mlsanity.com/paper/1605.09332/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 May 2016.

## Built on by (most cited first, 30 of 32)

- [Activation Functions in Deep Learning: A Comprehensive Survey and Benchmark](https://mlsanity.com/paper/2109.14545/) (2021)
- [How important are activation functions in regression and classification? A survey, performance comparison, and future directions](https://mlsanity.com/paper/2209.02681/) (2022)
- [Adaptive Rational Activations to Boost Deep Reinforcement Learning](https://mlsanity.com/paper/2102.09407/) (2021)
- [Neural Network Architecture Beyond Width and Depth](https://mlsanity.com/paper/2205.09459/) (2022)
- [Activation Functions in Artificial Neural Networks: A Systematic Overview](https://mlsanity.com/paper/2101.09957/) (2021)
- [Activation function design for deep networks: linearity and effective initialisation](https://mlsanity.com/paper/2105.07741/) (2021)
- [Rethinking Deep Thinking: Stable Learning of Algorithms using Lipschitz Constraints](https://mlsanity.com/paper/2410.23451/) (2024)
- [Reliable Detection of Compressed and Encrypted Data](https://mlsanity.com/paper/2103.17059/) (2021)
- [Improved weight initialization for deep and narrow feedforward neural network](https://mlsanity.com/paper/2311.03733/) (2023)
- [Maxwell's Demon at Work: Efficient Pruning by Leveraging Saturation of Neurons](https://mlsanity.com/paper/2403.07688/) (2024)
- [Trainable Highly-expressive Activation Functions](https://mlsanity.com/paper/2407.07564/) (2024)
- [A Generative Shape Compositional Framework to Synthesise Populations of Virtual Chimaeras](https://mlsanity.com/paper/2210.01607/) (2022)
- [Kolmogorov-Arnold Networks in Low-Data Regimes: A Comparative Study with Multilayer Perceptrons](https://mlsanity.com/paper/2409.10463/) (2024)
- [Why KDAC? A general activation function for knowledge discovery](https://mlsanity.com/paper/2111.13858/) (2021)
- [A Discriminative Single-Shot Segmentation Network for Visual Object Tracking](https://mlsanity.com/paper/2112.11846/) (2021)
- [Preprint: Norm Loss: An efficient yet effective regularization method for deep neural networks](https://mlsanity.com/paper/2103.06583/) (2021)
- [Adaptively Customizing Activation Functions for Various Layers](https://mlsanity.com/paper/2112.09442/) (2021)
- [Traffic incident duration prediction via a deep learning framework for text description encoding](https://mlsanity.com/paper/2209.08735/) (2022)
- [APALU: A Trainable, Adaptive Activation Function for Deep Learning Networks](https://mlsanity.com/paper/2402.08244/) (2024)
- [Preserving Plasticity in Continual Learning with Adaptive Linearity Injection](https://mlsanity.com/paper/2505.09486/) (2025)
- [Initializing ReLU networks in an expressive subspace of weights](https://mlsanity.com/paper/2103.12499/) (2021)
- [Piecewise Linear Units Improve Deep Neural Networks](https://mlsanity.com/paper/2108.00700/) (2021)
- [Sparse-Dyn: Sparse Dynamic Graph Multi-representation Learning via Event-based Sparse Temporal Attention Network](https://mlsanity.com/paper/2201.01384/) (2022)
- [Exploring the Relationship: Transformative Adaptive Activation Functions in Comparison to Other Activation Functions](https://mlsanity.com/paper/2402.09249/) (2024)
- [Efficient Search for Customized Activation Functions with Gradient Descent](https://mlsanity.com/paper/2408.06820/) (2024)
- [Unveiling the Potential of Superexpressive Networks in Implicit Neural Representations](https://mlsanity.com/paper/2503.21166/) (2025)
- [First Power Linear Unit with Sign](https://mlsanity.com/paper/2111.14349/) (2021)
- [Bayesian optimization for sparse neural networks with trainable activation functions](https://mlsanity.com/paper/2304.04455/) (2023)
- [Generalized Activation via Multivariate Projection](https://mlsanity.com/paper/2309.17194/) (2023)
- [Your Network May Need to Be Rewritten: Network Adversarial Based on High-Dimensional Function Graph Decomposition](https://mlsanity.com/paper/2405.03712/) (2024)

## Abstract (arXiv)

Object recognition is an important task for improving the ability of visual systems to perform complex scene understanding. Recently, the Exponential Linear Unit (ELU) has been proposed as a key component for managing bias shift in Convolutional Neural Networks (CNNs), but defines a parameter that must be set by hand. In this paper, we propose learning a parameterization of ELU in order to learn the proper activation shape at each layer in the CNNs. Our results on the MNIST, CIFAR-10/100 and ImageNet datasets using the NiN, Overfeat, All-CNN and ResNet networks indicate that our proposed Parametric ELU (PELU) has better performances than the non-parametric ELU. We have observed as much as a 7.28% relative error improvement on ImageNet with the NiN network, with only 0.0003% parameter increase. Our visual examination of the non-linear behaviors adopted by Vgg using PELU shows that the network took advantage of the added flexibility by learning different activations at different layers.