# Three Learning Stages and Accuracy-Efficiency Tradeoff of Restricted Boltzmann Machines

arXiv: https://arxiv.org/abs/2209.00873 · HTML: https://mlsanity.com/paper/2209.00873/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 72 references that are in the map.
- First posted on arXiv 2 Sep 2022.

## Builds on (most cited first, 16 of 16)

- [A mathematical theory of semantic development in deep neural networks](https://mlsanity.com/paper/1810.10531/) (2018)
- [A high-bias, low-variance introduction to Machine Learning for physicists](https://mlsanity.com/paper/1803.08823/) (2018)
- [Why does deep and cheap learning work so well?](https://mlsanity.com/paper/1608.08225/) (2016)
- [Application of Quantum Annealing to Training of Deep Neural Networks](https://mlsanity.com/paper/1510.06356/) (2015)
- [Mutual Information, Neural Networks and the Renormalization Group](https://mlsanity.com/paper/1704.06279/) (2017)
- [Neural-Network Quantum States, String-Bond States, and Chiral Topological States](https://mlsanity.com/paper/1710.04045/) (2017)
- [Equilibrium and non-Equilibrium regimes in the learning of Restricted Boltzmann Machines](https://mlsanity.com/paper/2105.13889/) (2021)
- [Restricted Boltzmann Machines: Introduction and Review](https://mlsanity.com/paper/1806.07066/) (2018)
- [Barriers and Dynamical Paths in Alternating Gibbs Sampling of Restricted Boltzmann Machines](https://mlsanity.com/paper/2107.06013/) (2021)
- [Accelerated physical emulation of Bayesian inference in spiking neural networks](https://mlsanity.com/paper/1807.02389/) (2018)
- [Stochastic inference with spiking neurons in the high-conductance state](https://mlsanity.com/paper/1610.07161/) (2016)
- [Hierarchical Models as Marginals of Hierarchical Models](https://mlsanity.com/paper/1508.03606/) (2015)
- [Accelerating Deep Learning with Memcomputing](https://mlsanity.com/paper/1801.00512/) (2018)
- [Spiking neuromorphic chip learns entangled quantum states](https://mlsanity.com/paper/2008.01039/) (2020)
- [Mode-Assisted Unsupervised Learning of Restricted Boltzmann Machines](https://mlsanity.com/paper/2001.05559/) (2020)
- [Variational learning of quantum ground states on spiking neuromorphic hardware](https://mlsanity.com/paper/2109.15169/) (2021)

## Built on by (most cited first, 2 of 2)

- [CMOS + stochastic nanomagnets: heterogeneous computers for probabilistic inference and learning](https://mlsanity.com/paper/2304.05949/) (2023)
- [Fast and Functional Structured Data Generators Rooted in Out-of-Equilibrium Physics](https://mlsanity.com/paper/2307.06797/) (2023)

## Abstract (arXiv)

Restricted Boltzmann Machines (RBMs) offer a versatile architecture for unsupervised machine learning that can in principle approximate any target probability distribution with arbitrary accuracy. However, the RBM model is usually not directly accessible due to its computational complexity, and Markov-chain sampling is invoked to analyze the learned probability distribution. For training and eventual applications, it is thus desirable to have a sampler that is both accurate and efficient. We highlight that these two goals generally compete with each other and cannot be achieved simultaneously. More specifically, we identify and quantitatively characterize three regimes of RBM learning: independent learning, where the accuracy improves without losing efficiency; correlation learning, where higher accuracy entails lower efficiency; and degradation, where both accuracy and efficiency no longer improve or even deteriorate. These findings are based on numerical experiments and heuristic arguments.