# Statistical mechanics of unsupervised feature learning in a restricted Boltzmann machine with binary synapses

arXiv: https://arxiv.org/abs/1612.01717 · HTML: https://mlsanity.com/paper/1612.01717/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Dec 2016.

## Built on by (most cited first, 10 of 10)

- [The emergence of a concept in shallow neural networks](https://mlsanity.com/paper/2109.00454/) (2021)
- [Learning a Restricted Boltzmann Machine using biased Monte Carlo sampling](https://mlsanity.com/paper/2206.01310/) (2022)
- [Hopfield model with planted patterns: a teacher-student self-supervised learning model](https://mlsanity.com/paper/2304.13710/) (2023)
- [Parallel Learning by Multitasking Neural Networks](https://mlsanity.com/paper/2308.04106/) (2023)
- [Effect of Weight Quantization on Learning Models by Typical Case Analysis](https://mlsanity.com/paper/2401.17269/) (2024)
- [The effect of priors on Learning with Restricted Boltzmann Machines](https://mlsanity.com/paper/2412.02623/) (2024)
- [Modeling Structured Data Learning with Restricted Boltzmann Machines in the Teacher-Student Setting](https://mlsanity.com/paper/2410.16150/) (2024)
- [Equivalence between algorithmic instability and transition to replica symmetry breaking in perceptron learning systems](https://mlsanity.com/paper/2111.13302/) (2021)
- [Learning with Restricted Boltzmann Machines: Asymptotics of AMP and GD in High Dimensions](https://mlsanity.com/paper/2505.18046/) (2025)
- [Saddle Hierarchy in Dense Associative Memory](https://mlsanity.com/paper/2508.19151/) (2025)

## Abstract (arXiv)

Revealing hidden features in unlabeled data is called unsupervised feature learning, which plays an important role in pretraining a deep neural network. Here we provide a statistical mechanics analysis of the unsupervised learning in a restricted Boltzmann machine with binary synapses. A message passing equation to infer the hidden feature is derived, and furthermore, variants of this equation are analyzed. A statistical analysis by replica theory describes the thermodynamic properties of the model. Our analysis confirms an entropy crisis preceding the non-convergence of the message passing equation, suggesting a discontinuous phase transition as a key characteristic of the restricted Boltzmann machine. Continuous phase transition is also confirmed depending on the embedded feature strength in the data. The mean-field result under the replica symmetric assumption agrees with that obtained by running message passing algorithms on single instances of finite sizes. Interestingly, in an approximate Hopfield model, the entropy crisis is absent, and a continuous phase transition is observed instead. We also develop an iterative equation to infer the hyper-parameter (temperature) hidden in the data, which in physics corresponds to iteratively imposing Nishimori condition. Our study provides insights towards understanding the thermodynamic properties of the restricted Boltzmann machine learning, and moreover important theoretical basis to build simplified deep networks.