# Universal Hopfield Networks: A General Framework for Single-Shot Associative Memory Models

arXiv: https://arxiv.org/abs/2202.04557 · HTML: https://mlsanity.com/paper/2202.04557/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 52 references that are in the map.
- First posted on arXiv 9 Feb 2022.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [Efficient Transformers: A Survey](https://mlsanity.com/paper/2009.06732/) (2020)
- [Hopfield Networks is All You Need](https://mlsanity.com/paper/2008.02217/) (2020)
- [Dense Associative Memory for Pattern Recognition](https://mlsanity.com/paper/1606.01164/) (2016)
- [Large Associative Memory Problem in Neurobiology and Machine Learning](https://mlsanity.com/paper/2008.06996/) (2020)
- [Augmenting Self-attention with Persistent Memory](https://mlsanity.com/paper/1907.01470/) (2019)
- [Memory-efficient Transformers via Top-k Attention](https://mlsanity.com/paper/2106.06899/) (2021)
- [Overparameterized Neural Networks Implement Associative Memory](https://mlsanity.com/paper/1909.12362/) (2019)
- [Associative Memories via Predictive Coding](https://mlsanity.com/paper/2109.08063/) (2021)
- [Hierarchical Associative Memory](https://mlsanity.com/paper/2107.06446/) (2021)
- [Attention Approximates Sparse Distributed Memory](https://mlsanity.com/paper/2111.05498/) (2021)
- [A remark on a paper of Krotov and Hopfield [arXiv:2008.06996]](https://mlsanity.com/paper/2105.15034/) (2021)
- [Associative Memory in Iterated Overparameterized Sigmoid Autoencoders](https://mlsanity.com/paper/2006.16540/) (2020)

## Built on by (most cited first, 30 of 55)

- [Test-time regression: a unifying framework for designing sequence models with associative memory](https://mlsanity.com/paper/2501.12352/) (2025)
- [On Sparse Modern Hopfield Model](https://mlsanity.com/paper/2309.12673/) (2023)
- [Uniform Memory Retrieval with Larger Capacity for Modern Hopfield Models](https://mlsanity.com/paper/2404.03827/) (2024)
- [STanHop: Sparse Tandem Hopfield Model for Memory-Enhanced Time Series Prediction](https://mlsanity.com/paper/2312.17346/) (2023)
- [Brain-inspired Computational Intelligence via Predictive Coding](https://mlsanity.com/paper/2308.07870/) (2023)
- [Fundamental Problems With Model Editing: How Should Rational Belief Revision Work in LLMs?](https://mlsanity.com/paper/2406.19354/) (2024)
- [Key-value memory in the brain](https://mlsanity.com/paper/2501.02950/) (2025)
- [Simplicial Hopfield networks](https://mlsanity.com/paper/2305.05179/) (2023)
- [Sequential Memory with Temporal Predictive Coding](https://mlsanity.com/paper/2305.11982/) (2023)
- [End-to-end Differentiable Clustering with Associative Memories](https://mlsanity.com/paper/2306.03209/) (2023)
- [Do LLMs dream of elephants (when told not to)? Latent concept association and associative memory in transformers](https://mlsanity.com/paper/2406.18400/) (2024)
- [BayesPCN: A Continually Learnable Predictive Coding Associative Memory](https://mlsanity.com/paper/2205.09930/) (2022)
- [Memory in Plain Sight: Surveying the Uncanny Resemblances of Associative Memories and Diffusion Models](https://mlsanity.com/paper/2309.16750/) (2023)
- [In-Context Exemplars as Clues to Retrieving from Large Associative Memory](https://mlsanity.com/paper/2311.03498/) (2023)
- [Sparse and Structured Hopfield Networks](https://mlsanity.com/paper/2402.13725/) (2024)
- [Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clusters](https://mlsanity.com/paper/2408.04093/) (2024)
- [Beyond Scaling Laws: Understanding Transformer Performance with Associative Memory](https://mlsanity.com/paper/2405.08707/) (2024)
- [Sparse Distributed Memory is a Continual Learner](https://mlsanity.com/paper/2303.11934/) (2023)
- [In-memory factorization of holographic perceptual representations](https://mlsanity.com/paper/2211.05052/) (2022)
- [Dense Associative Memory Through the Lens of Random Features](https://mlsanity.com/paper/2410.24153/) (2024)
- [Fast weight programming and linear transformers: from machine learning to neurobiology](https://mlsanity.com/paper/2508.08435/) (2025)
- [In-context denoising with one-layer transformers: connections between attention and associative memory retrieval](https://mlsanity.com/paper/2502.05164/) (2025)
- [A Sparse Quantized Hopfield Network for Online-Continual Memory](https://mlsanity.com/paper/2307.15040/) (2023)
- [Bridging Associative Memory and Probabilistic Modeling](https://mlsanity.com/paper/2402.10202/) (2024)
- [Hierarchical Associative Memory, Parallelized MLP-Mixer, and Symmetry Breaking](https://mlsanity.com/paper/2406.12220/) (2024)
- [On the Relationship Between Variational Inference and Auto-Associative Memory](https://mlsanity.com/paper/2210.08013/) (2022)
- [Zero-Shot Generalization of Vision-Based RL Without Data Augmentation](https://mlsanity.com/paper/2410.07441/) (2024)
- [An In-depth Investigation of Sparse Rate Reduction in Transformer-like Models](https://mlsanity.com/paper/2411.17182/) (2024)
- [Associative memory inspires improvements for in-context learning using a novel attention residual stream architecture](https://mlsanity.com/paper/2412.15113/) (2024)
- [DAM-Seg: Anatomically accurate cardiac segmentation using Dense Associative Networks](https://mlsanity.com/paper/2502.15128/) (2025)

## Abstract (arXiv)

A large number of neural network models of associative memory have been proposed in the literature. These include the classical Hopfield networks (HNs), sparse distributed memories (SDMs), and more recently the modern continuous Hopfield networks (MCHNs), which possesses close links with self-attention in machine learning. In this paper, we propose a general framework for understanding the operation of such memory networks as a sequence of three operations: similarity, separation, and projection. We derive all these memory models as instances of our general framework with differing similarity and separation functions. We extend the mathematical framework of Krotov et al (2020) to express general associative memory models using neural network dynamics with only second-order interactions between neurons, and derive a general energy function that is a Lyapunov function of the dynamics. Finally, using our framework, we empirically investigate the capacity of using different similarity functions for these associative memory models, beyond the dot product similarity measure, and demonstrate empirically that Euclidean or Manhattan distance similarity metrics perform substantially better in practice on many tasks, enabling a more robust retrieval and higher memory capacity than existing models.