# Stochastic Neural Networks with Monotonic Activation Functions

arXiv: https://arxiv.org/abs/1601.00034 · HTML: https://mlsanity.com/paper/1601.00034/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Jan 2016.

## Built on by (most cited first, 2 of 2)

- [Restricted Boltzmann Machines as Models of Interacting Variables](https://mlsanity.com/paper/2103.15917/) (2021)
- [Trainable Compound Activation Functions for Machine Learning](https://mlsanity.com/paper/2204.12920/) (2022)

## Abstract (arXiv)

We propose a Laplace approximation that creates a stochastic unit from any smooth monotonic activation function, using only Gaussian noise. This paper investigates the application of this stochastic approximation in training a family of Restricted Boltzmann Machines (RBM) that are closely linked to Bregman divergences. This family, that we call exponential family RBM (Exp-RBM), is a subset of the exponential family Harmoniums that expresses family members through a choice of smooth monotonic non-linearity for each neuron. Using contrastive divergence along with our Gaussian approximation, we show that Exp-RBM can learn useful representations using novel stochastic units.