# Neural Networks with A La Carte Selection of Activation Functions

arXiv: https://arxiv.org/abs/2206.12166 · HTML: https://mlsanity.com/paper/2206.12166/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 31 references that are in the map.
- First posted on arXiv 24 Jun 2022.

## Builds on (most cited first, 15 of 15)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [OpenML: networked science in machine learning](https://mlsanity.com/paper/1407.7722/) (2014)
- [ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functions](https://mlsanity.com/paper/2003.03488/) (2020)
- [Mish: A Self Regularized Non-Monotonic Activation Function](https://mlsanity.com/paper/1908.08681/) (2019)
- [Activation Functions: Comparison of trends in Practice and Research for Deep Learning](https://mlsanity.com/paper/1811.03378/) (2018)
- [A survey on modern trainable activation functions](https://mlsanity.com/paper/2005.00817/) (2020)
- [Learning Activation Functions to Improve Deep Neural Networks](https://mlsanity.com/paper/1412.6830/) (2014)
- [Noisy Activation Functions](https://mlsanity.com/paper/1603.00391/) (2016)
- [Smooth function approximation by deep neural networks with general activation functions](https://mlsanity.com/paper/1906.06903/) (2019)
- [SPLASH: Learnable Activation Functions for Improving Accuracy and Adversarial Robustness](https://mlsanity.com/paper/2006.08947/) (2020)
- [The Quest for the Golden Activation Function](https://mlsanity.com/paper/1808.00783/) (2018)
- [Ensemble of Convolutional Neural Networks Trained with Different Activation Functions](https://mlsanity.com/paper/1905.02473/) (2019)
- [Improved Adversarial Robustness by Reducing Open Space Risk via Tent Activations](https://mlsanity.com/paper/1908.02435/) (2019)

## Built on by (most cited first, 4 of 4)

- [Combining Deep Learning with Good Old-Fashioned Machine Learning](https://mlsanity.com/paper/2207.03757/) (2022)
- [High Per Parameter: A Large-Scale Study of Hyperparameter Tuning for Machine Learning Algorithms](https://mlsanity.com/paper/2207.06028/) (2022)
- [A Melting Pot of Evolution and Learning](https://mlsanity.com/paper/2306.04971/) (2023)
- [Hybrid Real- and Complex-valued Neural Network Architecture](https://mlsanity.com/paper/2504.03497/) (2025)

## Abstract (arXiv)

Activation functions (AFs), which are pivotal to the success (or failure) of a neural network, have received increased attention in recent years, with researchers seeking to design novel AFs that improve some aspect of network performance. In this paper we take another direction, wherein we combine a slew of known AFs into successful architectures, proposing three methods to do so beneficially: 1) generate AF architectures at random, 2) use Optuna, an automatic hyper-parameter optimization software framework, with a Tree-structured Parzen Estimator (TPE) sampler, and 3) use Optuna with a Covariance Matrix Adaptation Evolution Strategy (CMA-ES) sampler. We show that all methods often produce significantly better results for 25 classification problems when compared with a standard network composed of ReLU hidden units and a softmax output unit. Optuna with the TPE sampler emerged as the best AF architecture-producing method.