# Cascade of phase transitions in the training of Energy-based models

arXiv: https://arxiv.org/abs/2405.14689 · HTML: https://mlsanity.com/paper/2405.14689/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 52 references that are in the map.
- First posted on arXiv 23 May 2024.

## Builds on (most cited first, 17 of 17)

- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network](https://mlsanity.com/paper/1609.04802/) (2016)
- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [High-dimensional dynamics of generalization error in neural networks](https://mlsanity.com/paper/1710.03667/) (2017)
- [Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Model](https://mlsanity.com/paper/1904.09770/) (2019)
- [Dynamical Regimes of Diffusion Models](https://mlsanity.com/paper/2402.18491/) (2024)
- [A Phase Transition in Diffusion Models Reveals the Hierarchical Nature of Data](https://mlsanity.com/paper/2402.16991/) (2024)
- [Spontaneous Symmetry Breaking in Generative Diffusion Models](https://mlsanity.com/paper/2305.19693/) (2023)
- [Restricted Boltzmann Machine, recent advances and mean-field theory](https://mlsanity.com/paper/2011.11307/) (2020)
- [Equilibrium and non-Equilibrium regimes in the learning of Restricted Boltzmann Machines](https://mlsanity.com/paper/2105.13889/) (2021)
- [Thermodynamics of Restricted Boltzmann Machines and related learning dynamics](https://mlsanity.com/paper/1803.01960/) (2018)
- [Phase transitions and optimal algorithms in high-dimensional Gaussian mixture clustering](https://mlsanity.com/paper/1610.02918/) (2016)
- ['Place-cell' emergence and learning of invariant data with restricted Boltzmann machines: breaking and dynamical restoration of continuous symmetries in the weight space](https://mlsanity.com/paper/1912.12942/) (2019)
- [Cascade of Phase Transitions for Multi-Scale Clustering](https://mlsanity.com/paper/2010.07955/) (2020)
- [Unsupervised hierarchical clustering using the learning dynamics of RBMs](https://mlsanity.com/paper/2302.01851/) (2023)

## Built on by (most cited first, 12 of 12)

- [Fast training and sampling of Restricted Boltzmann Machines](https://mlsanity.com/paper/2405.15376/) (2024)
- [Local Diffusion Models and Phases of Data Distributions](https://mlsanity.com/paper/2508.06614/) (2025)
- [Fundamental operating regimes, hyper-parameter fine-tuning and glassiness: towards an interpretable replica-theory for trained restricted Boltzmann machines](https://mlsanity.com/paper/2406.09924/) (2024)
- [The effect of priors on Learning with Restricted Boltzmann Machines](https://mlsanity.com/paper/2412.02623/) (2024)
- [Steering Dynamical Regimes of Diffusion Models by Breaking Detailed Balance](https://mlsanity.com/paper/2602.15914/) (2026)
- [Modeling Structured Data Learning with Restricted Boltzmann Machines in the Teacher-Student Setting](https://mlsanity.com/paper/2410.16150/) (2024)
- [Learning with Restricted Boltzmann Machines: Asymptotics of AMP and GD in High Dimensions](https://mlsanity.com/paper/2505.18046/) (2025)
- [On the role of non-linear latent features in bipartite generative neural networks](https://mlsanity.com/paper/2506.10552/) (2025)
- [Generating configurations of increasing lattice size with machine learning and the inverse renormalization group](https://mlsanity.com/paper/2405.16288/) (2024)
- [Exploring the Energy Landscape of RBMs: Reciprocal Space Insights into Bosons, Hierarchical Learning and Symmetry Breaking](https://mlsanity.com/paper/2503.21536/) (2025)
- [Emergence of Nonequilibrium Latent Cycles in Unsupervised Generative Modeling](https://mlsanity.com/paper/2512.11415/) (2025)
- [Weighting Schedules Govern What and When Score-Based Generative Models Learn from Multimodal Data](https://mlsanity.com/paper/2609.35322/) (2026)

## Abstract (arXiv)

In this paper, we investigate the feature encoding process in a prototypical energy-based generative model, the Restricted Boltzmann Machine (RBM). We start with an analytical investigation using simplified architectures and data structures, and end with numerical analysis of real trainings on real datasets. Our study tracks the evolution of the model's weight matrix through its singular value decomposition, revealing a series of phase transitions associated to a progressive learning of the principal modes of the empirical probability distribution. The model first learns the center of mass of the modes and then progressively resolve all modes through a cascade of phase transitions. We first describe this process analytically in a controlled setup that allows us to study analytically the training dynamics. We then validate our theoretical results by training the Bernoulli-Bernoulli RBM on real data sets. By using data sets of increasing dimension, we show that learning indeed leads to sharp phase transitions in the high-dimensional limit. Moreover, we propose and test a mean-field finite-size scaling hypothesis. This shows that the first phase transition is in the same universality class of the one we studied analytically, and which is reminiscent of the mean-field paramagnetic-to-ferromagnetic phase transition.