# STL: A Signed and Truncated Logarithm Activation Function for Neural Networks

arXiv: https://arxiv.org/abs/2307.16389 · HTML: https://mlsanity.com/paper/2307.16389/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 50 references that are in the map.
- First posted on arXiv 31 Jul 2023.

## Builds on (most cited first, 17 of 17)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention](https://mlsanity.com/paper/2303.16199/) (2023)
- [Dynamic Neural Networks: A Survey](https://mlsanity.com/paper/2102.04906/) (2021)
- [Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing](https://mlsanity.com/paper/2306.12929/) (2023)
- [Training Language Models with Language Feedback at Scale](https://mlsanity.com/paper/2303.16755/) (2023)
- [Side Window Filtering](https://mlsanity.com/paper/1905.07177/) (2019)
- [What do Large Language Models Learn about Scripts?](https://mlsanity.com/paper/2112.13834/) (2021)
- [Weighted Mean Curvature](https://mlsanity.com/paper/1903.07189/) (2019)
- [Fast and High-Quality Blind Multi-Spectral Image Pansharpening](https://mlsanity.com/paper/2103.09943/) (2021)
- [A Discrete Scheme for Computing Image's Weighted Gaussian Curvature](https://mlsanity.com/paper/2101.07927/) (2021)
- [SERF: Towards better training of deep neural networks using log-Softplus ERror activation Function](https://mlsanity.com/paper/2108.09598/) (2021)
- [Quarter Laplacian Filter for Edge Aware Image Processing](https://mlsanity.com/paper/2101.07933/) (2021)
- [Natural-Logarithm-Rectified Activation Function in Convolutional Neural Networks](https://mlsanity.com/paper/1908.03682/) (2019)
- [Dynamic Large Language Models on Blockchains](https://mlsanity.com/paper/2307.10549/) (2023)
- [Multilevel Large Language Models for Everyone](https://mlsanity.com/paper/2307.13221/) (2023)

## Abstract (arXiv)

Activation functions play an essential role in neural networks. They provide the non-linearity for the networks. Therefore, their properties are important for neural networks' accuracy and running performance. In this paper, we present a novel signed and truncated logarithm function as activation function. The proposed activation function has significantly better mathematical properties, such as being odd function, monotone, differentiable, having unbounded value range, and a continuous nonzero gradient. These properties make it an excellent choice as an activation function. We compare it with other well-known activation functions in several well-known neural networks. The results confirm that it is the state-of-the-art. The suggested activation function can be applied in a large range of neural networks where activation functions are necessary.