# Parametric Leaky Tanh: A New Hybrid Activation Function for Deep Learning

arXiv: https://arxiv.org/abs/2310.07720 · HTML: https://mlsanity.com/paper/2310.07720/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 2 of its 7 references that are in the map.
- First posted on arXiv 11 Aug 2023.

## Builds on (most cited first, 2 of 2)

- [QReLU and m-QReLU: Two novel quantum activation functions to aid medical diagnostics](https://mlsanity.com/paper/2010.08031/) (2020)
- [ALReLU: A different approach on Leaky ReLU activation function to improve Neural Networks Performance](https://mlsanity.com/paper/2012.07564/) (2020)

## Built on by (most cited first, 1 of 1)

- [Zorro: A Flexible and Differentiable Parametric Family of Activation Functions That Extends ReLU and GELU](https://mlsanity.com/paper/2409.19239/) (2024)

## Abstract (arXiv)

Activation functions (AFs) are crucial components of deep neural networks (DNNs), having a significant impact on their performance. An activation function in a DNN is typically a smooth, nonlinear function that transforms an input signal into an output signal for the subsequent layer. In this paper, we propose the Parametric Leaky Tanh (PLTanh), a novel hybrid activation function designed to combine the strengths of both the Tanh and Leaky ReLU (LReLU) activation functions. PLTanh is differentiable at all points and addresses the 'dying ReLU' problem by ensuring a non-zero gradient for negative inputs, consistent with the behavior of LReLU. By integrating the unique advantages of these two diverse activation functions, PLTanh facilitates the learning of more intricate nonlinear relationships within the network. This paper presents an empirical evaluation of PLTanh against established activation functions, namely ReLU, LReLU, and ALReLU utilizing five diverse datasets.