mlsanityData to 7 Oct 2026

TSSR: A Truncated and Signed Square Root Activation Function for Neural Networks

arXiv 2308.04832 ↗First posted 9 Aug 2023cs.CV cs.CL cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Activation functions are essential components of neural networks. In this paper, we introduce a new activation function called the Truncated and Signed Square Root (TSSR) function. This function is distinctive because it is odd, nonlinear, monotone and differentiable. Its gradient is continuous and always positive. Thanks to these properties, it has the potential to improve the numerical stability of neural networks. Several experiments confirm that the proposed TSSR has better performance than other stat-of-the-art activation functions. The proposed function has significant implications for the development of neural network models and can be applied to a wide range of applications in fields such as computer vision, natural language processing, and speech recognition.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 17 of its 56 references that are in the map. First posted on arXiv 9 Aug 2023.

Builds on

17 of 56 references.

References · 17

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  3. 2023A Survey of Large Language Modelscited by 2,537
  4. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  5. 2021Dynamic Neural Networks: A Surveycited by 405
  6. 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingcited by 147
  7. 2023Training Language Models with Language Feedback at Scalecited by 125
  8. 2019Side Window Filteringcited by 27
  9. 2021What do Large Language Models Learn about Scripts?cited by 19
  10. 2019Weighted Mean Curvaturecited by 15
  11. 2021Fast and High-Quality Blind Multi-Spectral Image Pansharpeningcited by 13
  12. 2021A Discrete Scheme for Computing Image's Weighted Gaussian Curvaturecited by 10
5 more
  1. 2021SERF: Towards better training of deep neural networks using log-Softplus ERror activation Functioncited by 10
  2. 2021Quarter Laplacian Filter for Edge Aware Image Processingcited by 9
  3. 2023Flex-SFU: Accelerating DNN Activation Functions by Non-Uniform Piecewise Approximationcited by 8
  4. 2023Multilevel Large Language Models for Everyonecited by 5
  5. 2023Dynamic Large Language Models on Blockchainscited by 4

Built on by

Cited by year

Citing papers per year: 2023: 1.

1 paper.

  1. 2023PLMM: Personal Large Language Models on Mobile Devicescited by 0