# Fast training and sampling of Restricted Boltzmann Machines

arXiv: https://arxiv.org/abs/2405.15376 · HTML: https://mlsanity.com/paper/2405.15376/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 55 references that are in the map.
- First posted on arXiv 24 May 2024.

## Builds on (most cited first, 19 of 19)

- [How to Train Your Energy-Based Models](https://mlsanity.com/paper/2101.03288/) (2021)
- [A Theory of Generative ConvNet](https://mlsanity.com/paper/1602.03264/) (2016)
- [Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Model](https://mlsanity.com/paper/1904.09770/) (2019)
- [On the Anatomy of MCMC-Based Maximum Likelihood Learning of Energy-Based Models](https://mlsanity.com/paper/1903.12370/) (2019)
- [Inverse statistical problems: from the inverse Ising problem to data science](https://mlsanity.com/paper/1702.01522/) (2017)
- [Restricted Boltzmann Machine, recent advances and mean-field theory](https://mlsanity.com/paper/2011.11307/) (2020)
- [Equilibrium and non-Equilibrium regimes in the learning of Restricted Boltzmann Machines](https://mlsanity.com/paper/2105.13889/) (2021)
- [MCMC Should Mix: Learning Energy-Based Model with Neural Transport Latent Space MCMC](https://mlsanity.com/paper/2006.06897/) (2020)
- [Thermodynamics of Restricted Boltzmann Machines and related learning dynamics](https://mlsanity.com/paper/1803.01960/) (2018)
- [Gaussian-Bernoulli RBMs Without Tears](https://mlsanity.com/paper/2210.10318/) (2022)
- [Efficient Training of Energy-Based Models Using Jarzynski Equality](https://mlsanity.com/paper/2305.19414/) (2023)
- [Explaining the effects of non-convergent sampling in the training of Energy-Based Models](https://mlsanity.com/paper/2301.09428/) (2023)
- [Cascade of phase transitions in the training of Energy-based models](https://mlsanity.com/paper/2405.14689/) (2024)
- [Learning a Restricted Boltzmann Machine using biased Monte Carlo sampling](https://mlsanity.com/paper/2206.01310/) (2022)
- [Restricted Boltzmann Machines as Models of Interacting Variables](https://mlsanity.com/paper/2103.15917/) (2021)
- [Inferring effective couplings with Restricted Boltzmann Machines](https://mlsanity.com/paper/2309.02292/) (2023)
- [Fast and Functional Structured Data Generators Rooted in Out-of-Equilibrium Physics](https://mlsanity.com/paper/2307.06797/) (2023)
- [Balanced Training of Energy-Based Models with Adaptive Flow Sampling](https://mlsanity.com/paper/2306.00684/) (2023)
- [Population-Contrastive-Divergence: Does Consistency help with RBM training?](https://mlsanity.com/paper/1510.01624/) (2015)

## Built on by (most cited first, 9 of 9)

- [Learning Boltzmann Generators via Constrained Mass Transport](https://mlsanity.com/paper/2510.18460/) (2025)
- [Modeling Structured Data Learning with Restricted Boltzmann Machines in the Teacher-Student Setting](https://mlsanity.com/paper/2410.16150/) (2024)
- [Pseudo-likelihood produces associative memories able to generalize, even for asymmetric couplings](https://mlsanity.com/paper/2507.05147/) (2025)
- [On the role of non-linear latent features in bipartite generative neural networks](https://mlsanity.com/paper/2506.10552/) (2025)
- [PRIVET: PRoximIty leakage detection Via Extreme value Theory](https://mlsanity.com/paper/2510.24233/) (2025)
- [Emergence of Nonequilibrium Latent Cycles in Unsupervised Generative Modeling](https://mlsanity.com/paper/2512.11415/) (2025)
- [Distributional simplicity bias and effective convexity in Energy Based Models](https://mlsanity.com/paper/2605.07844/) (2026)
- [Explaining Machine Learning and Memorization with Statistical Mechanics](https://mlsanity.com/paper/2606.31110/) (2026)
- [Equilibrium Training of Energy-Based Models with Parallel Trajectory Tempering](https://mlsanity.com/paper/2607.27077/) (2026)

## Abstract (arXiv)

Restricted Boltzmann Machines (RBMs) are powerful tools for modeling complex systems and extracting insights from data, but their training is hindered by the slow mixing of Markov Chain Monte Carlo (MCMC) processes, especially with highly structured datasets. In this study, we build on recent theoretical advances in RBM training and focus on the stepwise encoding of data patterns into singular vectors of the coupling matrix, significantly reducing the cost of generating new samples and evaluating the quality of the model, as well as the training cost in highly clustered datasets. The learning process is analogous to the thermodynamic continuous phase transitions observed in ferromagnetic models, where new modes in the probability measure emerge in a continuous manner. We leverage the continuous transitions in the training process to define a smooth annealing trajectory that enables reliable and computationally efficient log-likelihood estimates. This approach enables online assessment during training and introduces a novel sampling strategy called Parallel Trajectory Tempering (PTT) that outperforms previously optimized MCMC methods. To mitigate the critical slowdown effect in the early stages of training, we propose a pre-training phase. In this phase, the principal components are encoded into a low-rank RBM through a convex optimization process, facilitating efficient static Monte Carlo sampling and accurate computation of the partition function. Our results demonstrate that this pre-training strategy allows RBMs to efficiently handle highly structured datasets where conventional methods fail. Additionally, our log-likelihood estimation outperforms computationally intensive approaches in controlled scenarios, while the PTT algorithm significantly accelerates MCMC processes compared to conventional methods.