mlsanityData to 7 Oct 2026

Modeling Structured Data Learning with Restricted Boltzmann Machines in the Teacher-Student Setting

arXiv 2410.16150 ↗First posted 21 Oct 2024cs.LG cond-mat.dis-nncited by 2

Not on the map. Why?

Abstract (arXiv)

Restricted Boltzmann machines (RBM) are generative models capable to learn data with a rich underlying structure. We study the teacher-student setting where a student RBM learns structured data generated by a teacher RBM. The amount of structure in the data is controlled by adjusting the number of hidden units of the teacher and the correlations in the rows of the weights, a.k.a. patterns. In the absence of correlations, we validate the conjecture that the performance is independent of the number of teacher patters and hidden units of the student RBMs, and we argue that the teacher-student setting can be used as a toy model for studying the lottery ticket hypothesis. Beyond this regime, we find that the critical amount of data required to learn the teacher patterns decreases with both their number and correlations. In both regimes, we find that, even with a relatively large dataset, it becomes impossible to learn the teacher patterns if the inference temperature used for regularization is kept too low. In our framework, the student can learn teacher patterns one-to-one or many-to-one, generalizing previous findings about the teacher-student setting with two hidden units to any arbitrary finite number of hidden units.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 42 of its 89 references that are in the map. First posted on arXiv 21 Oct 2024.

Builds on

42 of 89 references.

References · 42

  1. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  2. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  3. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  4. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  5. 2019What's Hidden in a Randomly Weighted Neural Network?cited by 236
  6. 2016Dense Associative Memory for Pattern Recognitioncited by 230
  7. 2019Deconstructing Lottery Tickets: Zeros, Signs, and the Supermaskcited by 206
  8. 2020Proving the Lottery Ticket Hypothesis: Pruning is All You Needcited by 200
  9. 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
  10. 2015High-Dimensional Asymptotics of Prediction: Ridge Regression and Classificationcited by 146
  11. 2020Optimal Regularization Can Mitigate Double Descentcited by 107
  12. 2020On the Optimal Weighted ℓ2 Regularization in Overparameterized Linear Regressioncited by 97
30 more
  1. 2015Optimal whitening and decorrelationcited by 93
  2. 2018Decorrelated Batch Normalizationcited by 83
  3. 2020A Random Matrix Analysis of Random Fourier Features: Beyond the Gaussian Kernel, a Precise Phase Transition, and the Corresponding Double Descentcited by 69
  4. 2021Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeedcited by 67
  5. 2020Phase diagram for two-layer ReLU neural networks at infinite-width limitcited by 65
  6. 2020Asymptotics of Ridge (less) Regression under General Source Conditioncited by 61
  7. 2021Learning Gaussian Mixtures with Generalised Linear Models: Precise Asymptotics in High-dimensionscited by 52
  8. 2020Multiple Descent: Design Your Own Generalization Curvecited by 44
  9. 2021Embedding Principle of Loss Landscape of Deep Neural Networkscited by 36
  10. 2020Restricted Boltzmann Machine, recent advances and mean-field theorycited by 31
  11. 2019Batch Normalization is a Cause of Adversarial Vulnerabilitycited by 28
  12. 2023Improved Discretization Analysis for Underdamped Langevin Monte Carlocited by 26
  13. 2017Phase Diagram of Restricted Boltzmann Machines and Generalised Hopfield Networks with Arbitrary Priorscited by 25
  14. 2018Thermodynamics of Restricted Boltzmann Machines and related learning dynamicscited by 25
  15. 2016Emergence of Compositional Representations in Restricted Boltzmann Machinescited by 23
  16. 2016Phase transitions in Restricted Boltzmann Machines with generic priorscited by 20
  17. 2019Minimal model of permutation symmetry in unsupervised learningcited by 13
  18. 2021On the interplay between data structure and loss function in classification problemscited by 12
  19. 2024Cascade of phase transitions in the training of Energy-based modelscited by 12
  20. 2016Statistical mechanics of unsupervised feature learning in a restricted Boltzmann machine with binary synapsescited by 10
  21. 2013Modeling Documents with Deep Boltzmann Machinescited by 9
  22. 2021Stochastic Whitening Batch Normalizationcited by 9
  23. 2024Fast training and sampling of Restricted Boltzmann Machinescited by 8
  24. 2020The Quenching-Activation Behavior of the Gradient Descent Dynamics for Two-layer Neural Network Modelscited by 7
  25. 2023Hopfield model with planted patterns: a teacher-student self-supervised learning modelcited by 7
  26. 2016Unsupervised feature learning from finite data by message passing: discontinuous versus continuous phase transitioncited by 6
  27. 2014From Dyson to Hopfield: Processing on hierarchical networkscited by 5
  28. 2024Dense Hopfield Networks in the Teacher-Student Settingcited by 4
  29. 2017Role of zero synapses in unsupervised feature learningcited by 3
  30. 2022Effect of Batch Normalization on Noise Resistant Property of Deep Learning Modelscited by 2

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2024The effect of priors on Learning with Restricted Boltzmann Machinescited by 3
  2. 2025Saddle Hierarchy in Dense Associative Memorycited by 1