# The Quest for the Golden Activation Function

arXiv: https://arxiv.org/abs/1808.00783 · HTML: https://mlsanity.com/paper/1808.00783/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Aug 2018.

## Built on by (most cited first, 14 of 14)

- [Activation Functions in Deep Learning: A Comprehensive Survey and Benchmark](https://mlsanity.com/paper/2109.14545/) (2021)
- [How important are activation functions in regression and classification? A survey, performance comparison, and future directions](https://mlsanity.com/paper/2209.02681/) (2022)
- [Activation Functions in Artificial Neural Networks: A Systematic Overview](https://mlsanity.com/paper/2101.09957/) (2021)
- [AutoInit: Analytic Signal-Preserving Weight Initialization for Neural Networks](https://mlsanity.com/paper/2109.08958/) (2021)
- [Efficient Activation Function Optimization through Surrogate Modeling](https://mlsanity.com/paper/2301.05785/) (2023)
- [Discovering Physics-Informed Neural Networks Model for Solving Partial Differential Equations through Evolutionary Computation](https://mlsanity.com/paper/2405.11208/) (2024)
- [Neural Networks with A La Carte Selection of Activation Functions](https://mlsanity.com/paper/2206.12166/) (2022)
- [Learning to Act through Evolution of Neural Diversity in Random Neural Networks](https://mlsanity.com/paper/2305.15945/) (2023)
- [Efficient Search for Customized Activation Functions with Gradient Descent](https://mlsanity.com/paper/2408.06820/) (2024)
- [Activation Function Optimization Scheme for Image Classification](https://mlsanity.com/paper/2409.04915/) (2024)
- [Research on Optimization Method of Multi-scale Fish Target Fast Detection Network](https://mlsanity.com/paper/2104.05050/) (2021)
- [First Power Linear Unit with Sign](https://mlsanity.com/paper/2111.14349/) (2021)
- [Nish: A Novel Negative Stimulated Hybrid Activation Function](https://mlsanity.com/paper/2210.09083/) (2022)
- [Bayesian optimization for sparse neural networks with trainable activation functions](https://mlsanity.com/paper/2304.04455/) (2023)

## Abstract (arXiv)

Deep Neural Networks have been shown to be beneficial for a variety of tasks, in particular allowing for end-to-end learning and reducing the requirement for manual design decisions. However, still many parameters have to be chosen in advance, also raising the need to optimize them. One important, but often ignored system parameter is the selection of a proper activation function. Thus, in this paper we target to demonstrate the importance of activation functions in general and show that for different tasks different activation functions might be meaningful. To avoid the manual design or selection of activation functions, we build on the idea of genetic algorithms to learn the best activation function for a given task. In addition, we introduce two new activation functions, ELiSH and HardELiSH, which can easily be incorporated in our framework. In this way, we demonstrate for three different image classification benchmarks that different activation functions are learned, also showing improved results compared to typically used baselines.