# Revise Saturated Activation Functions

arXiv: https://arxiv.org/abs/1602.05980 · HTML: https://mlsanity.com/paper/1602.05980/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Feb 2016.

## Built on by (most cited first, 11 of 11)

- [Activation Functions in Deep Learning: A Comprehensive Survey and Benchmark](https://mlsanity.com/paper/2109.14545/) (2021)
- [DeepDiagnosis: Automatically Diagnosing Faults and Recommending Actionable Fixes in Deep Learning Programs](https://mlsanity.com/paper/2112.04036/) (2021)
- [Making Pretrained Language Models Good Long-tailed Learners](https://mlsanity.com/paper/2205.05461/) (2022)
- [One model Packs Thousands of Items with Recurrent Conditional Query Learning](https://mlsanity.com/paper/2111.06726/) (2021)
- [Spiking Network Initialisation and Firing Rate Collapse](https://mlsanity.com/paper/2305.08879/) (2023)
- [S++: A Fast and Deployable Secure-Computation Framework for Privacy-Preserving Neural Network Training](https://mlsanity.com/paper/2101.12078/) (2021)
- [Exploring the Relationship: Transformative Adaptive Activation Functions in Comparison to Other Activation Functions](https://mlsanity.com/paper/2402.09249/) (2024)
- [LayerAct: Advanced Activation Mechanism for Robust Inference of CNNs](https://mlsanity.com/paper/2306.04940/) (2023)
- [A parametric activation function based on Wendland RBF](https://mlsanity.com/paper/2507.11493/) (2025)
- [Brownian ReLU(Br-ReLU): A New Activation Function for a Long-Short Term Memory (LSTM) Network](https://mlsanity.com/paper/2601.16446/) (2026)
- [Scaling Up Thermodynamic AI Models](https://mlsanity.com/paper/2607.00170/) (2026)

## Abstract (arXiv)

In this paper, we revise two commonly used saturated functions, the logistic sigmoid and the hyperbolic tangent (tanh). We point out that, besides the well-known non-zero centered property, slope of the activation function near the origin is another possible reason making training deep networks with the logistic function difficult to train. We demonstrate that, with proper rescaling, the logistic sigmoid achieves comparable results with tanh. Then following the same argument, we improve tahn by penalizing in the negative part. We show that "penalized tanh" is comparable and even outperforms the state-of-the-art non-saturated functions including ReLU and leaky ReLU on deep convolution neural networks. Our results contradict to the conclusion of previous works that the saturation property causes the slow convergence. It suggests further investigation is necessary to better understand activation functions in deep architectures.