# TanhSoft -- a family of activation functions combining Tanh and Softplus

arXiv: https://arxiv.org/abs/2009.03863 · HTML: https://mlsanity.com/paper/2009.03863/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Sep 2020.

## Built on by (most cited first, 2 of 2)

- [Orthogonal-Pad'e Activation Functions: Trainable Activation functions for smooth and faster convergence in deep networks](https://mlsanity.com/paper/2106.09693/) (2021)
- [A Family of Adaptive Activation Functions for Mitigating Failure Modes in Physics-Informed Neural Networks](https://mlsanity.com/paper/2603.18328/) (2026)

## Abstract (arXiv)

Deep learning at its core, contains functions that are composition of a linear transformation with a non-linear function known as activation function. In past few years, there is an increasing interest in construction of novel activation functions resulting in better learning. In this work, we propose a family of novel activation functions, namely TanhSoft, with four undetermined hyper-parameters of the form tanh(αx+βe^γx)ln(δ+ex) and tune these hyper-parameters to obtain activation functions which are shown to outperform several well known activation functions. For instance, replacing ReLU with xtanh(0.6ex)improves top-1 classification accuracy on CIFAR-10 by 0.46% for DenseNet-169 and 0.7% for Inception-v3 while with tanh(0.87x)ln(1 +ex) top-1 classification accuracy on CIFAR-100 improves by 1.24% for DenseNet-169 and 2.57% for SimpleNet model.