# PLU: The Piecewise Linear Unit Activation Function

arXiv: https://arxiv.org/abs/1809.09534 · HTML: https://mlsanity.com/paper/1809.09534/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Sep 2018.

## Built on by (most cited first, 11 of 11)

- [Robust Motion In-betweening](https://mlsanity.com/paper/2102.04942/) (2021)
- [Activation Functions in Deep Learning: A Comprehensive Survey and Benchmark](https://mlsanity.com/paper/2109.14545/) (2021)
- [Self-Supervised Nonlinear Transform-Based Tensor Nuclear Norm for Multi-Dimensional Image Recovery](https://mlsanity.com/paper/2105.14320/) (2021)
- [On the Omnipresence of Spurious Local Minima in Certain Neural Network Training Problems](https://mlsanity.com/paper/2202.12262/) (2022)
- [Optimal Activation Functions for the Random Features Regression Model](https://mlsanity.com/paper/2206.01332/) (2022)
- [Comparison of different convolutional neural network activation functions and methods for building ensembles](https://mlsanity.com/paper/2103.15898/) (2021)
- [Nonlinear Acoustic Echo Cancellation with Deep Learning](https://mlsanity.com/paper/2106.13754/) (2021)
- [Piecewise Linear Units Improve Deep Neural Networks](https://mlsanity.com/paper/2108.00700/) (2021)
- [Exploring the Relationship: Transformative Adaptive Activation Functions in Comparison to Other Activation Functions](https://mlsanity.com/paper/2402.09249/) (2024)
- [Neural networks with trainable matrix activation functions](https://mlsanity.com/paper/2109.09948/) (2021)
- [Optimizing Performance of Feedforward and Convolutional Neural Networks through Dynamic Activation Functions](https://mlsanity.com/paper/2308.05724/) (2023)

## Abstract (arXiv)

Successive linear transforms followed by nonlinear "activation" functions can approximate nonlinear functions to arbitrary precision given sufficient layers. The number of necessary layers is dependent on, in part, by the nature of the activation function. The hyperbolic tangent (tanh) has been a favorable choice as an activation until the networks grew deeper and the vanishing gradients posed a hindrance during training. For this reason the Rectified Linear Unit (ReLU) defined by max(0, x) has become the prevailing activation function in deep neural networks. Unlike the tanh function which is smooth, the ReLU yields networks that are piecewise linear functions with a limited number of facets. This paper presents a new activation function, the Piecewise Linear Unit (PLU) that is a hybrid of tanh and ReLU and shown to outperform the ReLU on a variety of tasks while avoiding the vanishing gradients issue of the tanh.