# Fundamental operating regimes, hyper-parameter fine-tuning and glassiness: towards an interpretable replica-theory for trained restricted Boltzmann machines

arXiv: https://arxiv.org/abs/2406.09924 · HTML: https://mlsanity.com/paper/2406.09924/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 80 references that are in the map.
- First posted on arXiv 14 Jun 2024.

## Builds on (most cited first, 18 of 18)

- [Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Kernel Renormalization](https://mlsanity.com/paper/2012.04030/) (2020)
- [Restricted Boltzmann Machine, recent advances and mean-field theory](https://mlsanity.com/paper/2011.11307/) (2020)
- [Equilibrium and non-Equilibrium regimes in the learning of Restricted Boltzmann Machines](https://mlsanity.com/paper/2105.13889/) (2021)
- [Phase Diagram of Restricted Boltzmann Machines and Generalised Hopfield Networks with Arbitrary Priors](https://mlsanity.com/paper/1702.05882/) (2017)
- [Thermodynamics of Restricted Boltzmann Machines and related learning dynamics](https://mlsanity.com/paper/1803.01960/) (2018)
- [Emergence of Compositional Representations in Restricted Boltzmann Machines](https://mlsanity.com/paper/1611.06759/) (2016)
- [Phase transitions in Restricted Boltzmann Machines with generic priors](https://mlsanity.com/paper/1612.03132/) (2016)
- [Learning Thermodynamics with Boltzmann Machines](https://mlsanity.com/paper/1606.02718/) (2016)
- [Dreaming neural networks: forgetting spurious memories and reinforcing pure ones](https://mlsanity.com/paper/1810.12217/) (2018)
- [Neural networks with redundant representation: detecting the undetectable](https://mlsanity.com/paper/1911.12689/) (2019)
- [Cascade of phase transitions in the training of Energy-based models](https://mlsanity.com/paper/2405.14689/) (2024)
- [Barriers and Dynamical Paths in Alternating Gibbs Sampling of Restricted Boltzmann Machines](https://mlsanity.com/paper/2107.06013/) (2021)
- [Disentangling representations in Restricted Boltzmann Machines without adversaries](https://mlsanity.com/paper/2206.11600/) (2022)
- [Modeling Documents with Deep Boltzmann Machines](https://mlsanity.com/paper/1309.6865/) (2013)
- [Advanced Mean Field Theory of Restricted Boltzmann Machine](https://mlsanity.com/paper/1502.00186/) (2015)
- [Mean-Field Inference in Gaussian Restricted Boltzmann Machine](https://mlsanity.com/paper/1512.00927/) (2015)
- [A Dynamical Mean-Field Theory for Learning in Restricted Boltzmann Machines](https://mlsanity.com/paper/2005.01560/) (2020)
- [Efficient training of energy-based models via spin-glass control](https://mlsanity.com/paper/1910.01592/) (2019)

## Built on by (most cited first, 3 of 3)

- [On the role of non-linear latent features in bipartite generative neural networks](https://mlsanity.com/paper/2506.10552/) (2025)
- [Activation Functions, Statistics and Learning of Higher-Order Interactions in Restricted Boltzmann Machines](https://mlsanity.com/paper/2605.19178/) (2026)
- [An Exploratory Replica-Overlap Probe of the Grokking Transition](https://mlsanity.com/paper/2609.25634/) (2026)

## Abstract (arXiv)

We consider restricted Boltzmann machines with a binary visible layer and a Gaussian hidden layer trained by an unlabelled dataset composed of noisy realizations of a single ground pattern. We develop a statistical mechanics framework to describe the network generative capabilities, by exploiting the replica trick and assuming self-averaging of the underlying order parameters (i.e., replica symmetry). In particular, we outline the effective control parameters (e.g., the relative number of weights to be trained, the regularization parameter), whose tuning can yield qualitatively-different operative regimes. Further, we provide analytical and numerical evidence for the existence of a sub-region in the space of the hyperparameters where replica-symmetry breaking occurs.