# Techniques for Learning Binary Stochastic Feedforward Neural Networks

arXiv: https://arxiv.org/abs/1406.2989 · HTML: https://mlsanity.com/paper/1406.2989/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jun 2014.

## Built on by (most cited first, 14 of 14)

- [A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classification](https://mlsanity.com/paper/2205.07877/) (2022)
- [Optimizing Language Models for Inference Time Objectives using Reinforcement Learning](https://mlsanity.com/paper/2503.19595/) (2025)
- [Selective compression learning of latent representations for variable-rate image compression](https://mlsanity.com/paper/2211.04104/) (2022)
- [Modelling Latent Translations for Cross-Lingual Transfer](https://mlsanity.com/paper/2107.11353/) (2021)
- [Unbiased Gradient Estimation with Balanced Assignments for Mixtures of Experts](https://mlsanity.com/paper/2109.11817/) (2021)
- [Fitting summary statistics of neural data with a differentiable spiking network simulator](https://mlsanity.com/paper/2106.10064/) (2021)
- [Bias-Variance Tradeoffs in Single-Sample Binary Gradient Estimators](https://mlsanity.com/paper/2110.03549/) (2021)
- [Learning Optimal Fronthauling and Decentralized Edge Computation in Fog Radio Access Networks](https://mlsanity.com/paper/2103.11284/) (2021)
- [Natural Quantization of Neural Networks](https://mlsanity.com/paper/2503.15482/) (2025)
- [AskewSGD : An Annealed interval-constrained Optimisation method to train Quantized Neural Networks](https://mlsanity.com/paper/2211.03741/) (2022)
- [Deep Neural Networks and End-to-End Learning for Audio Compression](https://mlsanity.com/paper/2105.11681/) (2021)
- [End-to-End Training for Back-Translation with Categorical Reparameterization Trick](https://mlsanity.com/paper/2202.08465/) (2022)
- [Learning to segment from object sizes](https://mlsanity.com/paper/2207.00289/) (2022)
- [Gibbs Sampling the Posterior of Neural Networks](https://mlsanity.com/paper/2306.02729/) (2023)

## Abstract (arXiv)

Stochastic binary hidden units in a multi-layer perceptron (MLP) network give at least three potential benefits when compared to deterministic MLP networks. (1) They allow to learn one-to-many type of mappings. (2) They can be used in structured prediction problems, where modeling the internal structure of the output is important. (3) Stochasticity has been shown to be an excellent regularizer, which makes generalization performance potentially better in general. However, training stochastic networks is considerably more difficult. We study training using M samples of hidden activations per input. We show that the case M=1 leads to a fundamentally different behavior where the network tries to avoid stochasticity. We propose two new estimators for the training gradient and propose benchmark tests for comparing training algorithms. Our experiments confirm that training stochastic networks is difficult and show that the proposed two estimators perform favorably among all the five known estimators.