# TSSR: A Truncated and Signed Square Root Activation Function for Neural Networks

arXiv: https://arxiv.org/abs/2308.04832 · HTML: https://mlsanity.com/paper/2308.04832/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 56 references that are in the map.
- First posted on arXiv 9 Aug 2023.

## Builds on (most cited first, 17 of 17)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention](https://mlsanity.com/paper/2303.16199/) (2023)
- [Dynamic Neural Networks: A Survey](https://mlsanity.com/paper/2102.04906/) (2021)
- [Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing](https://mlsanity.com/paper/2306.12929/) (2023)
- [Training Language Models with Language Feedback at Scale](https://mlsanity.com/paper/2303.16755/) (2023)
- [Side Window Filtering](https://mlsanity.com/paper/1905.07177/) (2019)
- [What do Large Language Models Learn about Scripts?](https://mlsanity.com/paper/2112.13834/) (2021)
- [Weighted Mean Curvature](https://mlsanity.com/paper/1903.07189/) (2019)
- [Fast and High-Quality Blind Multi-Spectral Image Pansharpening](https://mlsanity.com/paper/2103.09943/) (2021)
- [A Discrete Scheme for Computing Image's Weighted Gaussian Curvature](https://mlsanity.com/paper/2101.07927/) (2021)
- [SERF: Towards better training of deep neural networks using log-Softplus ERror activation Function](https://mlsanity.com/paper/2108.09598/) (2021)
- [Quarter Laplacian Filter for Edge Aware Image Processing](https://mlsanity.com/paper/2101.07933/) (2021)
- [Flex-SFU: Accelerating DNN Activation Functions by Non-Uniform Piecewise Approximation](https://mlsanity.com/paper/2305.04546/) (2023)
- [Dynamic Large Language Models on Blockchains](https://mlsanity.com/paper/2307.10549/) (2023)
- [Multilevel Large Language Models for Everyone](https://mlsanity.com/paper/2307.13221/) (2023)

## Built on by (most cited first, 1 of 1)

- [PLMM: Personal Large Language Models on Mobile Devices](https://mlsanity.com/paper/2309.14726/) (2023)

## Abstract (arXiv)

Activation functions are essential components of neural networks. In this paper, we introduce a new activation function called the Truncated and Signed Square Root (TSSR) function. This function is distinctive because it is odd, nonlinear, monotone and differentiable. Its gradient is continuous and always positive. Thanks to these properties, it has the potential to improve the numerical stability of neural networks. Several experiments confirm that the proposed TSSR has better performance than other stat-of-the-art activation functions. The proposed function has significant implications for the development of neural network models and can be applied to a wide range of applications in fields such as computer vision, natural language processing, and speech recognition.