# Exploring the Temperature-Dependent Phase Transition in Modern Hopfield Networks

arXiv: https://arxiv.org/abs/2311.18434 · HTML: https://mlsanity.com/paper/2311.18434/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 20 references that are in the map.
- First posted on arXiv 30 Nov 2023.

## Builds on (most cited first, 7 of 7)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Hopfield Networks is All You Need](https://mlsanity.com/paper/2008.02217/) (2020)
- [A Survey of Transformers](https://mlsanity.com/paper/2106.04554/) (2021)
- [Dense Associative Memory for Pattern Recognition](https://mlsanity.com/paper/1606.01164/) (2016)
- [Energy Transformer](https://mlsanity.com/paper/2302.07253/) (2023)
- [Towards Biologically Plausible Deep Learning](https://mlsanity.com/paper/1502.04156/) (2015)
- [Dense Associative Memory is Robust to Adversarial Inputs](https://mlsanity.com/paper/1701.00939/) (2017)

## Built on by (most cited first, 1 of 1)

- [Phases in a class of associative memories via hidden neurons](https://mlsanity.com/paper/2609.10976/) (2026)

## Abstract (arXiv)

The recent discovery of a connection between Transformers and Modern Hopfield Networks (MHNs) has reignited the study of neural networks from a physical energy-based perspective. This paper focuses on the pivotal effect of the inverse temperature hyperparameter β on the distribution of energy minima of the MHN. To achieve this, the distribution of energy minima is tracked in a simplified MHN in which equidistant normalised patterns are stored. This network demonstrates a phase transition at a critical temperature βc, from a single global attractor towards highly pattern specific minima as β is increased. Importantly, the dynamics are not solely governed by the hyperparameter β but are instead determined by an effective inverse temperature βeff which also depends on the distribution and size of the stored patterns. Recognizing the role of hyperparameters in the MHN could, in the future, aid researchers in the domain of Transformers to optimise their initial choices, potentially reducing the necessity for time and energy expensive hyperparameter fine-tuning.