# A Biologically Plausible Dense Associative Memory with Exponential Capacity

arXiv: https://arxiv.org/abs/2601.00984 · HTML: https://mlsanity.com/paper/2601.00984/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Jan 2026.

## Built on by (most cited first, 6 of 6)

- [On the Infinite Width and Depth Limits of Predictive Coding Networks](https://mlsanity.com/paper/2602.07697/) (2026)
- [Energy-Based Dynamical Models for Neurocomputation, Learning, and Optimization](https://mlsanity.com/paper/2604.05042/) (2026)
- [Algorithmic Analysis of Dense Associative Memory: Finite-Size Guarantees and Adversarial Robustness](https://mlsanity.com/paper/2604.12811/) (2026)
- [Context-Gated Associative Retrieval: From Theory to Transformers](https://mlsanity.com/paper/2605.10970/) (2026)
- [Phases in a class of associative memories via hidden neurons](https://mlsanity.com/paper/2609.10976/) (2026)
- [Hierarchical Prototype Emergence in Modern Hopfield Models](https://mlsanity.com/paper/2609.12079/) (2026)

## Abstract (arXiv)

Krotov and Hopfield (2021) proposed a biologically plausible two-layer associative memory network with memory storage capacity exponential in the number of visible neurons. However, the capacity was only linear in the number of hidden neurons. This limitation arose from the choice of nonlinearity between the visible and hidden units, which enforced winner-take-all dynamics in the hidden layer, thereby restricting each hidden unit to encode only a single memory. We overcome this limitation by introducing a novel associative memory network with a threshold nonlinearity that enables distributed representations. In contrast to winner-take-all dynamics, where each hidden neuron is tied to an entire memory, our network allows hidden neurons to encode basic components shared across many memories. Consequently, complex patterns are represented through combinations of hidden neurons. These representations reduce redundancy and allow many correlated memories to be stored compositionally. Thus, we achieve much higher capacity: exponential in the number of hidden units, provided the number of visible units is sufficiently large relative to the number of hidden units. Exponential capacity arises because all binary states of the hidden units can become stable memory patterns. Moreover, the distributed hidden representation, which has much lower dimensionality than the visible layer, preserves class-discriminative structure, supporting efficient nonlinear decoding. These results establish a new regime for associative memory, enabling high-capacity, robust, and scalable architectures consistent with biological constraints.