# Emergence of Compositional Representations in Restricted Boltzmann Machines

arXiv: https://arxiv.org/abs/1611.06759 · HTML: https://mlsanity.com/paper/1611.06759/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Nov 2016.

## Built on by (most cited first, 23 of 23)

- [The emergence of a concept in shallow neural networks](https://mlsanity.com/paper/2109.00454/) (2021)
- [Learning a Restricted Boltzmann Machine using biased Monte Carlo sampling](https://mlsanity.com/paper/2206.01310/) (2022)
- [Disentangling representations in Restricted Boltzmann Machines without adversaries](https://mlsanity.com/paper/2206.11600/) (2022)
- [Restricted Boltzmann Machines as Models of Interacting Variables](https://mlsanity.com/paper/2103.15917/) (2021)
- [Inferring effective couplings with Restricted Boltzmann Machines](https://mlsanity.com/paper/2309.02292/) (2023)
- [Fast and Functional Structured Data Generators Rooted in Out-of-Equilibrium Physics](https://mlsanity.com/paper/2307.06797/) (2023)
- [Quantifying Relevance in Learning and Inference](https://mlsanity.com/paper/2202.00339/) (2022)
- [Unsupervised hierarchical clustering using the learning dynamics of RBMs](https://mlsanity.com/paper/2302.01851/) (2023)
- [GenEFT: Understanding Statics and Dynamics of Model Generalization via Effective Theory](https://mlsanity.com/paper/2402.05916/) (2024)
- [Interpretable machine learning of amino acid patterns in proteins: a statistical ensemble approach](https://mlsanity.com/paper/2303.15228/) (2023)
- [Parallel Learning by Multitasking Neural Networks](https://mlsanity.com/paper/2308.04106/) (2023)
- [Fundamental operating regimes, hyper-parameter fine-tuning and glassiness: towards an interpretable replica-theory for trained restricted Boltzmann machines](https://mlsanity.com/paper/2406.09924/) (2024)
- [Compositionally Equivariant Representation Learning](https://mlsanity.com/paper/2306.07783/) (2023)
- [Modeling Structured Data Learning with Restricted Boltzmann Machines in the Teacher-Student Setting](https://mlsanity.com/paper/2410.16150/) (2024)
- [A developmental approach for training deep belief networks](https://mlsanity.com/paper/2207.05473/) (2022)
- [Statistical Mechanics of Learning via Reverberation in Bidirectional Associative Memories](https://mlsanity.com/paper/2307.08365/) (2023)
- [On the role of non-linear latent features in bipartite generative neural networks](https://mlsanity.com/paper/2506.10552/) (2025)
- [Pattern reconstruction with restricted Boltzmann machines](https://mlsanity.com/paper/2205.07087/) (2022)
- [Unsupervised and Supervised learning by Dense Associative Memory under replica symmetry breaking](https://mlsanity.com/paper/2312.09638/) (2023)
- [Configuration Interaction Guided Sampling with Interpretable Restricted Boltzmann Machine](https://mlsanity.com/paper/2409.06146/) (2024)
- [Emergence of Nonequilibrium Latent Cycles in Unsupervised Generative Modeling](https://mlsanity.com/paper/2512.11415/) (2025)
- [Activation Functions, Statistics and Learning of Higher-Order Interactions in Restricted Boltzmann Machines](https://mlsanity.com/paper/2605.19178/) (2026)
- [Equilibrium Training of Energy-Based Models with Parallel Trajectory Tempering](https://mlsanity.com/paper/2607.27077/) (2026)

## Abstract (arXiv)

Extracting automatically the complex set of features composing real high-dimensional data is crucial for achieving high performance in machine--learning tasks. Restricted Boltzmann Machines (RBM) are empirically known to be efficient for this purpose, and to be able to generate distributed and graded representations of the data. We characterize the structural conditions (sparsity of the weights, low effective temperature, nonlinearities in the activation functions of hidden units, and adaptation of fields maintaining the activity in the visible layer) allowing RBM to operate in such a compositional phase. Evidence is provided by the replica analysis of an adequate statistical ensemble of random RBMs and by RBM trained on the handwritten digits dataset MNIST.