# Is it Time to Swish? Comparing Deep Learning Activation Functions Across NLP tasks

arXiv: https://arxiv.org/abs/1901.02671 · HTML: https://mlsanity.com/paper/1901.02671/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jan 2019.

## Built on by (most cited first, 9 of 9)

- [Activation Functions in Deep Learning: A Comprehensive Survey and Benchmark](https://mlsanity.com/paper/2109.14545/) (2021)
- [How important are activation functions in regression and classification? A survey, performance comparison, and future directions](https://mlsanity.com/paper/2209.02681/) (2022)
- [Wide and Deep Neural Networks Achieve Optimality for Classification](https://mlsanity.com/paper/2204.14126/) (2022)
- [A fast accurate fine-grain object detection model based on YOLOv4 deep neural network](https://mlsanity.com/paper/2111.00298/) (2021)
- [Simple and complex spiking neurons: perspectives and analysis in a simple STDP scenario](https://mlsanity.com/paper/2207.04881/) (2022)
- [A Critical Review of Physics-Informed Machine Learning Applications in Subsurface Energy Systems](https://mlsanity.com/paper/2308.04457/) (2023)
- [Your Network May Need to Be Rewritten: Network Adversarial Based on High-Dimensional Function Graph Decomposition](https://mlsanity.com/paper/2405.03712/) (2024)
- [Swish-T : Enhancing Swish Activation with Tanh Bias for Improved Neural Network Performance](https://mlsanity.com/paper/2407.01012/) (2024)
- [LingVarBench: Benchmarking LLMs on Entity Recognitions and Linguistic Verbalization Patterns in Phone-Call Transcripts](https://mlsanity.com/paper/2508.15801/) (2025)

## Abstract (arXiv)

Activation functions play a crucial role in neural networks because they are the nonlinearities which have been attributed to the success story of deep learning. One of the currently most popular activation functions is ReLU, but several competitors have recently been proposed or 'discovered', including LReLU functions and swish. While most works compare newly proposed activation functions on few tasks (usually from image classification) and against few competitors (usually ReLU), we perform the first large-scale comparison of 21 activation functions across eight different NLP tasks. We find that a largely unknown activation function performs most stably across all tasks, the so-called penalized tanh function. We also show that it can successfully replace the sigmoid and tanh gates in LSTM cells, leading to a 2 percentage point (pp) improvement over the standard choices on a challenging NLP task.