Learning with Restricted Boltzmann Machines: Asymptotics of AMP and GD in High Dimensions
Not on the map. Why?
Abstract (arXiv)
The Restricted Boltzmann Machine (RBM) is one of the simplest generative neural networks capable of learning input distributions. Despite its simplicity, the analysis of its performance in learning from the training data is only well understood in cases that essentially reduce to singular value decomposition of the data. Here, we consider the limit of a large dimension of the input space and a constant number of hidden units. In this limit, we simplify the standard RBM training objective into a form that is equivalent to the multi-index model with non-separable regularization. This opens a path to analyze training of the RBM using methods that are established for multi-index models, such as Approximate Message Passing (AMP) and its state evolution, and the analysis of Gradient Descent (GD) via the dynamical mean-field theory. We then give rigorous asymptotics of the training dynamics of RBM on data generated by the spiked covariance model as a prototype of a structure suitable for unsupervised learning. We show in particular that RBM reaches the optimal computational weak recovery threshold, aligning with the BBP transition, in the spiked covariance model.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 34 of its 73 references that are in the map. First posted on arXiv 23 May 2025.
Builds on
34 of 73 references.
References · 34
- 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
- 2023SGD learning on neural networks: leap complexity and saddle-to-saddle dynamicscited by 143
- 2017Theoretical insights into the optimization landscape of over-parameterized shallow neural networkscited by 128
- 2014A statistical model for tensor PCAcited by 72
- 2023Smoothing the Landscape Boosts the Signal for SGD: Optimal Sample Complexity for Learning Single Index Modelscited by 61
- 2020Dynamical mean-field theory for stochastic gradient descent in Gaussian mixture classificationcited by 60
- 2023On Learning Gaussian Multi-index Models with Gradient Flowcited by 57
- 2017Estimation of Low-Rank Matrices via Approximate Message Passingcited by 41
- 2017The landscape of the spiked tensor modelcited by 34
- 2020The estimation error of general first order methodscited by 33
- 2020Approximate Message Passing algorithms for rotationally invariant matricescited by 32
- 2020Restricted Boltzmann Machine, recent advances and mean-field theorycited by 31
22 more
- 2017Phase Diagram of Restricted Boltzmann Machines and Generalised Hopfield Networks with Arbitrary Priorscited by 25
- 2018Thermodynamics of Restricted Boltzmann Machines and related learning dynamicscited by 25
- 2020Generalization Error of Generalized Linear Models in High Dimensionscited by 25
- 2021The effective noise of Stochastic Gradient Descentcited by 25
- 2020Asymptotic Errors for Teacher-Student Convex Generalized Linear Models (or : How to Prove Kabashima's Replica Formula)cited by 23
- 2018Fundamental limits of detection in the spiked Wigner modelcited by 18
- 2021Approximate Message Passing for orthogonally invariant ensembles: Multivariate non-linearities and spectral initializationcited by 18
- 2019Inference with Deep Generative Priors in High Dimensionscited by 17
- 2020Universality of Approximate Message Passing Algorithmscited by 16
- 2025Asymptotics of Non-Convex Generalized Linear Models in High-Dimensions: A proof of the replica formulacited by 14
- 2019Minimal model of permutation symmetry in unsupervised learningcited by 13
- 2024Cascade of phase transitions in the training of Energy-based modelscited by 12
- 2025Fundamental Limits of Matrix Sensing: Exact Asymptotics, Universality, and Applicationscited by 11
- 2016Statistical mechanics of unsupervised feature learning in a restricted Boltzmann machine with binary synapsescited by 10
- 2019'Place-cell' emergence and learning of invariant data with restricted Boltzmann machines: breaking and dynamical restoration of continuous symmetries in the weight spacecited by 10
- 2023Approximate message passing from random initialization with applications to mathbbZ2 synchronizationcited by 9
- 2023Hopfield model with planted patterns: a teacher-student self-supervised learning modelcited by 7
- 2016Unsupervised feature learning from finite data by message passing: discontinuous versus continuous phase transitioncited by 6
- 2017A Deterministic and Generalized Framework for Unsupervised Learning with Restricted Boltzmann Machinescited by 6
- 2024The effect of priors on Learning with Restricted Boltzmann Machinescited by 3
- 2019Gaussian-Spherical Restricted Boltzmann Machinescited by 2
- 2024Fundamental limits of Non-Linear Low-Rank Matrix Estimationcited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.