# Bayesian optimization for sparse neural networks with trainable activation functions

arXiv: https://arxiv.org/abs/2304.04455 · HTML: https://mlsanity.com/paper/2304.04455/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 76 references that are in the map.
- First posted on arXiv 10 Apr 2023.

## Builds on (most cited first, 19 of 19)

- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Image Super-Resolution Using Deep Convolutional Networks](https://mlsanity.com/paper/1501.00092/) (2014)
- [Self-Normalizing Neural Networks](https://mlsanity.com/paper/1706.02515/) (2017)
- [Deep Learning for Image Super-resolution: A Survey](https://mlsanity.com/paper/1902.06068/) (2019)
- [A survey on modern trainable activation functions](https://mlsanity.com/paper/2005.00817/) (2020)
- [Noisy Activation Functions](https://mlsanity.com/paper/1603.00391/) (2016)
- [A Comprehensive Analysis of Deep Regression](https://mlsanity.com/paper/1803.08450/) (2018)
- [Parametric Exponential Linear Unit for Deep Convolutional Neural Networks](https://mlsanity.com/paper/1605.09332/) (2016)
- [Complex-valued Neural Networks with Non-parametric Activation Functions](https://mlsanity.com/paper/1802.08026/) (2018)
- [Zero-bias autoencoders and the benefits of co-adapting features](https://mlsanity.com/paper/1402.3337/) (2014)
- [The Quest for the Golden Activation Function](https://mlsanity.com/paper/1808.00783/) (2018)
- [Learning activation functions from data using cubic spline interpolation](https://mlsanity.com/paper/1605.05509/) (2016)
- [Adaptive Blending Units: Trainable Activation Functions for Deep Neural Networks](https://mlsanity.com/paper/1806.10064/) (2018)
- [Deep Residual Networks with Exponential Linear Unit](https://mlsanity.com/paper/1604.04112/) (2016)
- [Ensemble of Convolutional Neural Networks Trained with Different Activation Functions](https://mlsanity.com/paper/1905.02473/) (2019)
- [Flatten-T Swish: a thresholded ReLU-Swish-like activation function for deep learning](https://mlsanity.com/paper/1812.06247/) (2018)
- [Evolution of Activation Functions: An Empirical Investigation](https://mlsanity.com/paper/2105.14614/) (2021)
- [A linear approach for sparse coding by a two-layer neural network](https://mlsanity.com/paper/1507.01892/) (2015)

## Abstract (arXiv)

In the literature on deep neural networks, there is considerable interest in developing activation functions that can enhance neural network performance. In recent years, there has been renewed scientific interest in proposing activation functions that can be trained throughout the learning process, as they appear to improve network performance, especially by reducing overfitting. In this paper, we propose a trainable activation function whose parameters need to be estimated. A fully Bayesian model is developed to automatically estimate from the learning data both the model weights and activation function parameters. An MCMC-based optimization scheme is developed to build the inference. The proposed method aims to solve the aforementioned problems and improve convergence time by using an efficient sampling scheme that guarantees convergence to the global maximum. The proposed scheme is tested on three datasets with three different CNNs. Promising results demonstrate the usefulness of our proposed approach in improving model accuracy due to the proposed activation function and Bayesian estimation of the parameters.