mlsanityData to 7 Oct 2026

Parametric Exponential Linear Unit for Deep Convolutional Neural Networks

arXiv 1605.09332 ↗First posted 30 May 2016cs.LG cs.CV cs.NEcited by 32

Not on the map. Why?

Abstract (arXiv)

Object recognition is an important task for improving the ability of visual systems to perform complex scene understanding. Recently, the Exponential Linear Unit (ELU) has been proposed as a key component for managing bias shift in Convolutional Neural Networks (CNNs), but defines a parameter that must be set by hand. In this paper, we propose learning a parameterization of ELU in order to learn the proper activation shape at each layer in the CNNs. Our results on the MNIST, CIFAR-10/100 and ImageNet datasets using the NiN, Overfeat, All-CNN and ResNet networks indicate that our proposed Parametric ELU (PELU) has better performances than the non-parametric ELU. We have observed as much as a 7.28% relative error improvement on ImageNet with the NiN network, with only 0.0003% parameter increase. Our visual examination of the non-linear behaviors adopted by Vgg using PELU shows that the network took advantage of the added flexibility by learning different activations at different layers.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). First posted on arXiv 30 May 2016.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 12; 2022: 5; 2023: 3; 2024: 8; 2025: 4.

32 papers.

Sort:

  1. 2021Activation Functions in Deep Learning: A Comprehensive Survey and Benchmarkcited by 153
  2. 2022How important are activation functions in regression and classification? A survey, performance comparison, and future directionscited by 29
  3. 2021Adaptive Rational Activations to Boost Deep Reinforcement Learningcited by 23
  4. 2022Neural Network Architecture Beyond Width and Depthcited by 17
  5. 2021Activation function design for deep networks: linearity and effective initialisationcited by 13
  6. 2021Activation Functions in Artificial Neural Networks: A Systematic Overviewcited by 12
  7. 2024Rethinking Deep Thinking: Stable Learning of Algorithms using Lipschitz Constraintscited by 8
  8. 2021Reliable Detection of Compressed and Encrypted Datacited by 7
  9. 2023Improved weight initialization for deep and narrow feedforward neural networkcited by 7
  10. 2024Trainable Highly-expressive Activation Functionscited by 6
  11. 2022A Generative Shape Compositional Framework to Synthesise Populations of Virtual Chimaerascited by 5
  12. 2024Maxwell's Demon at Work: Efficient Pruning by Leveraging Saturation of Neuronscited by 5
20 more
  1. 2024Kolmogorov-Arnold Networks in Low-Data Regimes: A Comparative Study with Multilayer Perceptronscited by 5
  2. 2021Why KDAC? A general activation function for knowledge discoverycited by 4
  3. 2021A Discriminative Single-Shot Segmentation Network for Visual Object Trackingcited by 4
  4. 2021Preprint: Norm Loss: An efficient yet effective regularization method for deep neural networkscited by 3
  5. 2021Adaptively Customizing Activation Functions for Various Layerscited by 3
  6. 2022Traffic incident duration prediction via a deep learning framework for text description encodingcited by 3
  7. 2024APALU: A Trainable, Adaptive Activation Function for Deep Learning Networkscited by 3
  8. 2021Initializing ReLU networks in an expressive subspace of weightscited by 1
  9. 2021Piecewise Linear Units Improve Deep Neural Networkscited by 1
  10. 2022Sparse-Dyn: Sparse Dynamic Graph Multi-representation Learning via Event-based Sparse Temporal Attention Networkcited by 1
  11. 2024Exploring the Relationship: Transformative Adaptive Activation Functions in Comparison to Other Activation Functionscited by 1
  12. 2024Efficient Search for Customized Activation Functions with Gradient Descentcited by 1
  13. 2025Unveiling the Potential of Superexpressive Networks in Implicit Neural Representationscited by 1
  14. 2025Preserving Plasticity in Continual Learning with Adaptive Linearity Injectioncited by 1
  15. 2021First Power Linear Unit with Signcited by 0
  16. 2023Bayesian optimization for sparse neural networks with trainable activation functionscited by 0
  17. 2023Generalized Activation via Multivariate Projectioncited by 0
  18. 2024Your Network May Need to Be Rewritten: Network Adversarial Based on High-Dimensional Function Graph Decompositioncited by 0
  19. 2025Exploring the Feasibility of Deep Learning Techniques for Accurate Gender Classification from Eye Imagescited by 0
  20. 2025ULU: A Unified Activation Functioncited by 0