# On the role of synaptic stochasticity in training low-precision neural networks

arXiv: https://arxiv.org/abs/1710.09825 · HTML: https://mlsanity.com/paper/1710.09825/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Oct 2017.

## Built on by (most cited first, 4 of 4)

- [Deep Networks on Toroids: Removing Symmetries Reveals the Structure of Flat Regions in the Landscape Geometry](https://mlsanity.com/paper/2202.03038/) (2022)
- [Statistical mechanics of continual learning: variational principle and mean-field potential](https://mlsanity.com/paper/2212.02846/) (2022)
- [HEMP: High-order Entropy Minimization for neural network comPression](https://mlsanity.com/paper/2107.05298/) (2021)
- [Equivalence between algorithmic instability and transition to replica symmetry breaking in perceptron learning systems](https://mlsanity.com/paper/2111.13302/) (2021)

## Abstract (arXiv)

Stochasticity and limited precision of synaptic weights in neural network models are key aspects of both biological and hardware modeling of learning processes. Here we show that a neural network model with stochastic binary weights naturally gives prominence to exponentially rare dense regions of solutions with a number of desirable properties such as robustness and good generalization performance, while typical solutions are isolated and hard to find. Binary solutions of the standard perceptron problem are obtained from a simple gradient descent procedure on a set of real values parametrizing a probability distribution over the binary synapses. Both analytical and numerical results are presented. An algorithmic extension aimed at training discrete deep neural networks is also investigated.