# Attention in a family of Boltzmann machines emerging from modern Hopfield networks

arXiv: https://arxiv.org/abs/2212.04692 · HTML: https://mlsanity.com/paper/2212.04692/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 24 references that are in the map.
- First posted on arXiv 9 Dec 2022.

## Builds on (most cited first, 7 of 7)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [Hopfield Networks is All You Need](https://mlsanity.com/paper/2008.02217/) (2020)
- [Dense Associative Memory for Pattern Recognition](https://mlsanity.com/paper/1606.01164/) (2016)
- [Large Associative Memory Problem in Neurobiology and Machine Learning](https://mlsanity.com/paper/2008.06996/) (2020)
- [Gaussian-Bernoulli RBMs Without Tears](https://mlsanity.com/paper/2210.10318/) (2022)
- [A remark on a paper of Krotov and Hopfield [arXiv:2008.06996]](https://mlsanity.com/paper/2105.15034/) (2021)

## Built on by (most cited first, 7 of 7)

- [Hierarchical Associative Memory, Parallelized MLP-Mixer, and Symmetry Breaking](https://mlsanity.com/paper/2406.12220/) (2024)
- [Saddle Hierarchy in Dense Associative Memory](https://mlsanity.com/paper/2508.19151/) (2025)
- [A Boltzmann-machine-enhanced Transformer For DNA Sequence Classification](https://mlsanity.com/paper/2603.26465/) (2026)
- [Continual Learning in Modern Hopfield Networks with an Application to Diffusion Models](https://mlsanity.com/paper/2605.27975/) (2026)
- [Boltzmann Attention: Learnable Ising Couplings for Cooperative Attention](https://mlsanity.com/paper/2606.12478/) (2026)
- [Variational-Ising-Attention:Tailored Attention Matters for Science](https://mlsanity.com/paper/2607.23634/) (2026)
- [Phases in a class of associative memories via hidden neurons](https://mlsanity.com/paper/2609.10976/) (2026)

## Abstract (arXiv)

Hopfield networks and Boltzmann machines (BMs) are fundamental energy-based neural network models. Recent studies on modern Hopfield networks have broaden the class of energy functions and led to a unified perspective on general Hopfield networks including an attention module. In this letter, we consider the BM counterparts of modern Hopfield networks using the associated energy functions, and study their salient properties from a trainability perspective. In particular, the energy function corresponding to the attention module naturally introduces a novel BM, which we refer to as the attentional BM (AttnBM). We verify that AttnBM has a tractable likelihood function and gradient for certain special cases and is easy to train. Moreover, we reveal the hidden connections between AttnBM and some single-layer models, namely the Gaussian--Bernoulli restricted BM and the denoising autoencoder with softmax units coming from denoising score matching. We also investigate BMs introduced by other energy functions and show that the energy function of dense associative memory models gives BMs belonging to Exponential Family Harmoniums.