# Restricted Boltzmann Machine and Deep Belief Network: Tutorial and Survey

arXiv: https://arxiv.org/abs/2107.12521 · HTML: https://mlsanity.com/paper/2107.12521/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 66 references that are in the map.
- First posted on arXiv 26 Jul 2021.

## Builds on (most cited first, 11 of 11)

- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [Hopfield Networks is All You Need](https://mlsanity.com/paper/2008.02217/) (2020)
- [Dense Associative Memory for Pattern Recognition](https://mlsanity.com/paper/1606.01164/) (2016)
- [Log-concave sampling: Metropolis-Hastings algorithms are fast](https://mlsanity.com/paper/1801.02309/) (2018)
- [Large Associative Memory Problem in Neurobiology and Machine Learning](https://mlsanity.com/paper/2008.06996/) (2020)
- [Hierarchical Associative Memory](https://mlsanity.com/paper/2107.06446/) (2021)
- [The Theory Behind Overfitting, Cross Validation, Regularization, Bagging, and Boosting: Tutorial](https://mlsanity.com/paper/1905.12787/) (2019)
- [Modeling Documents with Deep Boltzmann Machines](https://mlsanity.com/paper/1309.6865/) (2013)
- [Sampling Algorithms, from Survey Sampling to Monte Carlo Methods: Tutorial and Literature Review](https://mlsanity.com/paper/2011.00901/) (2020)
- [Temperature-Based Deep Boltzmann Machines](https://mlsanity.com/paper/1608.07719/) (2016)
- [Marginal Densities, Factor Graph Duality, and High-Temperature Series Expansions](https://mlsanity.com/paper/1901.02733/) (2019)

## Built on by (most cited first, 5 of 5)

- [Spectral, Probabilistic, and Deep Metric Learning: Tutorial and Survey](https://mlsanity.com/paper/2201.09267/) (2022)
- [Wavefunction Flows: Efficient Quantum Simulation of Continuous Flow Models](https://mlsanity.com/paper/2510.08462/) (2025)
- [Neural Boltzmann Machines](https://mlsanity.com/paper/2305.08337/) (2023)
- [Quantitative Universal Approximation Bounds for Deep Belief Networks](https://mlsanity.com/paper/2208.09033/) (2022)
- [Towards Unsupervised Representation Learning: Learning, Evaluating and Transferring Visual Representations](https://mlsanity.com/paper/2312.00101/) (2023)

## Abstract (arXiv)

This is a tutorial and survey paper on Boltzmann Machine (BM), Restricted Boltzmann Machine (RBM), and Deep Belief Network (DBN). We start with the required background on probabilistic graphical models, Markov random field, Gibbs sampling, statistical physics, Ising model, and the Hopfield network. Then, we introduce the structures of BM and RBM. The conditional distributions of visible and hidden variables, Gibbs sampling in RBM for generating variables, training BM and RBM by maximum likelihood estimation, and contrastive divergence are explained. Then, we discuss different possible discrete and continuous distributions for the variables. We introduce conditional RBM and how it is trained. Finally, we explain deep belief network as a stack of RBM models. This paper on Boltzmann machines can be useful in various fields including data science, statistics, neural computation, and statistical physics.