# Dense Hopfield Networks in the Teacher-Student Setting

arXiv: https://arxiv.org/abs/2401.04191 · HTML: https://mlsanity.com/paper/2401.04191/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 28 of its 83 references that are in the map.
- First posted on arXiv 8 Jan 2024.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 28 of 28)

- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Adversarial Examples Are Not Bugs, They Are Features](https://mlsanity.com/paper/1905.02175/) (2019)
- [Robustness May Be at Odds with Accuracy](https://mlsanity.com/paper/1805.12152/) (2018)
- [Evasion Attacks against Machine Learning at Test Time](https://mlsanity.com/paper/1708.06131/) (2017)
- [Hopfield Networks is All You Need](https://mlsanity.com/paper/2008.02217/) (2020)
- [Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examples](https://mlsanity.com/paper/2010.03593/) (2020)
- [Dense Associative Memory for Pattern Recognition](https://mlsanity.com/paper/1606.01164/) (2016)
- [A Universal Law of Robustness via Isoperimetry](https://mlsanity.com/paper/2105.12806/) (2021)
- [Statistical physics of inference: Thresholds and algorithms](https://mlsanity.com/paper/1511.02476/) (2015)
- [Large Associative Memory Problem in Neurobiology and Machine Learning](https://mlsanity.com/paper/2008.06996/) (2020)
- [Exploring Architectural Ingredients of Adversarially Robust Deep Neural Networks](https://mlsanity.com/paper/2110.03825/) (2021)
- [Energy Transformer](https://mlsanity.com/paper/2302.07253/) (2023)
- [A law of robustness for two-layers neural networks](https://mlsanity.com/paper/2009.14444/) (2020)
- [Modern Hopfield Networks and Attention for Immune Repertoire Classification](https://mlsanity.com/paper/2007.13505/) (2020)
- [Dense Associative Memory is Robust to Adversarial Inputs](https://mlsanity.com/paper/1701.00939/) (2017)
- [In search of dispersed memories: Generative diffusion models are associative memory networks](https://mlsanity.com/paper/2309.17290/) (2023)
- [On the Adversarial Robustness of Mixture of Experts](https://mlsanity.com/paper/2210.10253/) (2022)
- [Phase Diagram of Restricted Boltzmann Machines and Generalised Hopfield Networks with Arbitrary Priors](https://mlsanity.com/paper/1702.05882/) (2017)
- [Overparameterized Linear Regression under Adversarial Attacks](https://mlsanity.com/paper/2204.06274/) (2022)
- [Phase transitions in Restricted Boltzmann Machines with generic priors](https://mlsanity.com/paper/1612.03132/) (2016)
- [With Friends Like These, Who Needs Adversaries?](https://mlsanity.com/paper/1807.04200/) (2018)
- [Minimal model of permutation symmetry in unsupervised learning](https://mlsanity.com/paper/1904.13052/) (2019)
- [Neural networks with redundant representation: detecting the undetectable](https://mlsanity.com/paper/1911.12689/) (2019)
- [Glassy nature of the hard phase in inference problems](https://mlsanity.com/paper/1805.05857/) (2018)
- [Hopfield model with planted patterns: a teacher-student self-supervised learning model](https://mlsanity.com/paper/2304.13710/) (2023)
- [A Waddington landscape for prototype learning in generalized Hopfield networks](https://mlsanity.com/paper/2312.03012/) (2023)

## Built on by (most cited first, 4 of 4)

- [The effect of priors on Learning with Restricted Boltzmann Machines](https://mlsanity.com/paper/2412.02623/) (2024)
- [Modeling Structured Data Learning with Restricted Boltzmann Machines in the Teacher-Student Setting](https://mlsanity.com/paper/2410.16150/) (2024)
- [Saddle Hierarchy in Dense Associative Memory](https://mlsanity.com/paper/2508.19151/) (2025)
- [Temporal Complexity and Self-Organization in an Exponential Dense Associative Memory Model](https://mlsanity.com/paper/2601.11478/) (2026)

## Abstract (arXiv)

Dense Hopfield networks are known for their feature to prototype transition and adversarial robustness. However, previous theoretical studies have been mostly concerned with their storage capacity. We bridge this gap by studying the phase diagram of p-body Hopfield networks in the teacher-student setting of an unsupervised learning problem, uncovering ferromagnetic phases reminiscent of the prototype and feature learning regimes. On the Nishimori line, we find the critical size of the training set necessary for efficient pattern retrieval. Interestingly, we find that that the paramagnetic to ferromagnetic transition of the teacher-student setting coincides with the paramagnetic to spin-glass transition of the direct model, i.e. with random patterns. Outside of the Nishimori line, we investigate the learning performance in relation to the inference temperature and dataset noise. Moreover, we show that using a larger p for the student than the teacher gives the student an extensive tolerance to noise. We then derive a closed-form expression measuring the adversarial robustness of such a student at zero temperature, corroborating the positive correlation between number of parameters and robustness observed in large neural networks. We also use our model to clarify why the prototype phase of modern Hopfield networks is adversarially robust.