# Thermodynamics of Restricted Boltzmann Machines and related learning dynamics

arXiv: https://arxiv.org/abs/1803.01960 · HTML: https://mlsanity.com/paper/1803.01960/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Mar 2018.

## Built on by (most cited first, 25 of 25)

- [Equilibrium and non-Equilibrium regimes in the learning of Restricted Boltzmann Machines](https://mlsanity.com/paper/2105.13889/) (2021)
- [The emergence of a concept in shallow neural networks](https://mlsanity.com/paper/2109.00454/) (2021)
- [Explaining the effects of non-convergent sampling in the training of Energy-Based Models](https://mlsanity.com/paper/2301.09428/) (2023)
- [Cascade of phase transitions in the training of Energy-based models](https://mlsanity.com/paper/2405.14689/) (2024)
- [Barriers and Dynamical Paths in Alternating Gibbs Sampling of Restricted Boltzmann Machines](https://mlsanity.com/paper/2107.06013/) (2021)
- [Learning a Restricted Boltzmann Machine using biased Monte Carlo sampling](https://mlsanity.com/paper/2206.01310/) (2022)
- [Fast training and sampling of Restricted Boltzmann Machines](https://mlsanity.com/paper/2405.15376/) (2024)
- [Inferring effective couplings with Restricted Boltzmann Machines](https://mlsanity.com/paper/2309.02292/) (2023)
- [Fast and Functional Structured Data Generators Rooted in Out-of-Equilibrium Physics](https://mlsanity.com/paper/2307.06797/) (2023)
- [Quantifying Relevance in Learning and Inference](https://mlsanity.com/paper/2202.00339/) (2022)
- [High-dimensional Asymptotics of VAEs: Threshold of Posterior Collapse and Dataset-Size Dependence of Rate-Distortion Curve](https://mlsanity.com/paper/2309.07663/) (2023)
- [A theoretical framework for overfitting in energy-based modeling](https://mlsanity.com/paper/2501.19158/) (2025)
- [Unsupervised hierarchical clustering using the learning dynamics of RBMs](https://mlsanity.com/paper/2302.01851/) (2023)
- [Interpretable machine learning of amino acid patterns in proteins: a statistical ensemble approach](https://mlsanity.com/paper/2303.15228/) (2023)
- [Fundamental operating regimes, hyper-parameter fine-tuning and glassiness: towards an interpretable replica-theory for trained restricted Boltzmann machines](https://mlsanity.com/paper/2406.09924/) (2024)
- [The effect of priors on Learning with Restricted Boltzmann Machines](https://mlsanity.com/paper/2412.02623/) (2024)
- [Recurrent neural networks that generalize from examples and optimize by dreaming](https://mlsanity.com/paper/2204.07954/) (2022)
- [Modeling Structured Data Learning with Restricted Boltzmann Machines in the Teacher-Student Setting](https://mlsanity.com/paper/2410.16150/) (2024)
- [A simple probabilistic neural network for machine understanding](https://mlsanity.com/paper/2210.13179/) (2022)
- [Statistical Mechanics of Min-Max Problems](https://mlsanity.com/paper/2409.06053/) (2024)
- [Learning with Restricted Boltzmann Machines: Asymptotics of AMP and GD in High Dimensions](https://mlsanity.com/paper/2505.18046/) (2025)
- [On the role of non-linear latent features in bipartite generative neural networks](https://mlsanity.com/paper/2506.10552/) (2025)
- [Saddle Hierarchy in Dense Associative Memory](https://mlsanity.com/paper/2508.19151/) (2025)
- [Exploring the Energy Landscape of RBMs: Reciprocal Space Insights into Bosons, Hierarchical Learning and Symmetry Breaking](https://mlsanity.com/paper/2503.21536/) (2025)
- [Equilibrium Training of Energy-Based Models with Parallel Trajectory Tempering](https://mlsanity.com/paper/2607.27077/) (2026)

## Abstract (arXiv)

We investigate the thermodynamic properties of a Restricted Boltzmann Machine (RBM), a simple energy-based generative model used in the context of unsupervised learning. Assuming the information content of this model to be mainly reflected by the spectral properties of its weight matrix W, we try to make a realistic analysis by averaging over an appropriate statistical ensemble of RBMs. First, a phase diagram is derived. Otherwise similar to that of the Sherrington- Kirkpatrick (SK) model with ferromagnetic couplings, the RBM's phase diagram presents a ferromagnetic phase which may or may not be of compositional type depending on the kurtosis of the distribution of the components of the singular vectors of W. Subsequently, the learning dynamics of the RBM is studied in the thermodynamic limit. A "typical" learning trajectory is shown to solve an effective dynamical equation, based on the aforementioned ensemble average and explicitly involving order parameters obtained from the thermodynamic analysis. In particular, this let us show how the evolution of the dominant singular values of W, and thus of the unstable modes, is driven by the input data. At the beginning of the training, in which the RBM is found to operate in the linear regime, the unstable modes reflect the dominant covariance modes of the data. In the non-linear regime, instead, the selected modes interact and eventually impose a matching of the order parameters to their empirical counterparts estimated from the data. Finally, we illustrate our considerations by performing experiments on both artificial and real data, showing in particular how the RBM operates in the ferromagnetic compositional phase.