# Learning a Restricted Boltzmann Machine using biased Monte Carlo sampling

arXiv: https://arxiv.org/abs/2206.01310 · HTML: https://mlsanity.com/paper/2206.01310/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 44 references that are in the map.
- First posted on arXiv 2 Jun 2022.

## Builds on (most cited first, 10 of 10)

- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Restricted Boltzmann Machine, recent advances and mean-field theory](https://mlsanity.com/paper/2011.11307/) (2020)
- [Equilibrium and non-Equilibrium regimes in the learning of Restricted Boltzmann Machines](https://mlsanity.com/paper/2105.13889/) (2021)
- [Phase Diagram of Restricted Boltzmann Machines and Generalised Hopfield Networks with Arbitrary Priors](https://mlsanity.com/paper/1702.05882/) (2017)
- [Thermodynamics of Restricted Boltzmann Machines and related learning dynamics](https://mlsanity.com/paper/1803.01960/) (2018)
- [Emergence of Compositional Representations in Restricted Boltzmann Machines](https://mlsanity.com/paper/1611.06759/) (2016)
- [Barriers and Dynamical Paths in Alternating Gibbs Sampling of Restricted Boltzmann Machines](https://mlsanity.com/paper/2107.06013/) (2021)
- [Statistical mechanics of unsupervised feature learning in a restricted Boltzmann machine with binary synapses](https://mlsanity.com/paper/1612.01717/) (2016)
- ['Place-cell' emergence and learning of invariant data with restricted Boltzmann machines: breaking and dynamical restoration of continuous symmetries in the weight space](https://mlsanity.com/paper/1912.12942/) (2019)
- [Learning Compositional Representations of Interacting Systems with Restricted Boltzmann Machines: Comparative Study of Lattice Proteins](https://mlsanity.com/paper/1902.06495/) (2019)

## Built on by (most cited first, 11 of 11)

- [Explaining the effects of non-convergent sampling in the training of Energy-Based Models](https://mlsanity.com/paper/2301.09428/) (2023)
- [Fast training and sampling of Restricted Boltzmann Machines](https://mlsanity.com/paper/2405.15376/) (2024)
- [Inferring effective couplings with Restricted Boltzmann Machines](https://mlsanity.com/paper/2309.02292/) (2023)
- [Fast and Functional Structured Data Generators Rooted in Out-of-Equilibrium Physics](https://mlsanity.com/paper/2307.06797/) (2023)
- [A theoretical framework for overfitting in energy-based modeling](https://mlsanity.com/paper/2501.19158/) (2025)
- [Unsupervised hierarchical clustering using the learning dynamics of RBMs](https://mlsanity.com/paper/2302.01851/) (2023)
- [Balanced Training of Energy-Based Models with Adaptive Flow Sampling](https://mlsanity.com/paper/2306.00684/) (2023)
- [Parallel Learning by Multitasking Neural Networks](https://mlsanity.com/paper/2308.04106/) (2023)
- [On the role of non-linear latent features in bipartite generative neural networks](https://mlsanity.com/paper/2506.10552/) (2025)
- [Configuration Interaction Guided Sampling with Interpretable Restricted Boltzmann Machine](https://mlsanity.com/paper/2409.06146/) (2024)
- [Equilibrium Training of Energy-Based Models with Parallel Trajectory Tempering](https://mlsanity.com/paper/2607.27077/) (2026)

## Abstract (arXiv)

Restricted Boltzmann Machines are simple and powerful generative models that can encode any complex dataset. Despite all their advantages, in practice the trainings are often unstable and it is difficult to assess their quality because the dynamics are affected by extremely slow time dependencies. This situation becomes critical when dealing with low-dimensional clustered datasets, where the time required to sample ergodically the trained models becomes computationally prohibitive. In this work, we show that this divergence of Monte Carlo mixing times is related to a phenomenon of phase coexistence, similar to that which occurs in physics near a first-order phase transition. We show that sampling the equilibrium distribution using the Markov chain Monte Carlo method can be dramatically accelerated when using biased sampling techniques, in particular the Tethered Monte Carlo (TMC) method. This sampling technique efficiently solves the problem of evaluating the quality of a given trained model and generating new samples in a reasonable amount of time. Moreover, we show that this sampling technique can also be used to improve the computation of the log-likelihood gradient during training, leading to dramatic improvements in training RBMs with artificial clustered datasets. On real low-dimensional datasets, this new training method fits RBM models with significantly faster relaxation dynamics than those obtained with standard PCD recipes. We also show that TMC sampling can be used to recover the free-energy profile of the RBM. This proves to be extremely useful to compute the probability distribution of a given model and to improve the generation of new decorrelated samples in slow PCD-trained models.