# ProbAct: A Probabilistic Activation Function for Deep Neural Networks

arXiv: https://arxiv.org/abs/1905.10761 · HTML: https://mlsanity.com/paper/1905.10761/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 May 2019.

## Built on by (most cited first, 12 of 12)

- [Activation Functions in Deep Learning: A Comprehensive Survey and Benchmark](https://mlsanity.com/paper/2109.14545/) (2021)
- [Deep Kronecker neural networks: A general framework for neural networks with adaptive activation functions](https://mlsanity.com/paper/2105.09513/) (2021)
- [How important are activation functions in regression and classification? A survey, performance comparison, and future directions](https://mlsanity.com/paper/2209.02681/) (2022)
- [SERF: Towards better training of deep neural networks using log-Softplus ERror activation Function](https://mlsanity.com/paper/2108.09598/) (2021)
- [Panacea: Novel DNN Accelerator using Accuracy-Preserving Asymmetric Quantization and Energy-Saving Bit-Slice Sparsity](https://mlsanity.com/paper/2412.10059/) (2024)
- [BubbleOKAN: A Physics-Informed Interpretable Neural Operator for High-Frequency Bubble Dynamics](https://mlsanity.com/paper/2508.03965/) (2025)
- [Simpler Certified Radius Maximization by Propagating Covariances](https://mlsanity.com/paper/2104.05888/) (2021)
- [Langevin algorithms for Markovian Neural Networks and Deep Stochastic control](https://mlsanity.com/paper/2212.12018/) (2022)
- [Density Regression and Uncertainty Quantification with Bayesian Deep Noise Neural Networks](https://mlsanity.com/paper/2206.05643/) (2022)
- [Langevin algorithms for very deep Neural Networks with application to image classification](https://mlsanity.com/paper/2212.14718/) (2022)
- [Generation of Realistic Cloud Access Times for Mobile Application Testing using Transfer Learning](https://mlsanity.com/paper/2103.09355/) (2021)
- [BrowNNe: Brownian Nonlocal Neurons & Activation Functions](https://mlsanity.com/paper/2406.15617/) (2024)

## Abstract (arXiv)

Activation functions play an important role in training artificial neural networks. The majority of currently used activation functions are deterministic in nature, with their fixed input-output relationship. In this work, we propose a novel probabilistic activation function, called ProbAct. ProbAct is decomposed into a mean and variance and the output value is sampled from the formed distribution, making ProbAct a stochastic activation function. The values of mean and variances can be fixed using known functions or trained for each element. In the trainable ProbAct, the mean and the variance of the activation distribution is trained within the back-propagation framework alongside other parameters. We show that the stochastic perturbation induced through ProbAct acts as a viable generalization technique for feature augmentation. In our experiments, we compare ProbAct with well-known activation functions on classification tasks on different modalities: Images(CIFAR-10, CIFAR-100, and STL-10) and Text (Large Movie Review). We show that ProbAct increases the classification accuracy by +2-3% compared to ReLU or other conventional activation functions on both original datasets and when datasets are reduced to 50% and 25% of the original size. Finally, we show that ProbAct learns an ensemble of models by itself that can be used to estimate the uncertainties associated with the prediction and provides robustness to noisy inputs.