# Exponential Capacity in Multilayer Hetero-Associative Neural Networks

arXiv: https://arxiv.org/abs/2607.29554 · HTML: https://mlsanity.com/paper/2607.29554/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 52 references that are in the map.
- First posted on arXiv 31 Jul 2026.

## Builds on (most cited first, 16 of 16)

- [Hopfield Networks is All You Need](https://mlsanity.com/paper/2008.02217/) (2020)
- [An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction](https://mlsanity.com/paper/1909.02027/) (2019)
- [Dense Associative Memory for Pattern Recognition](https://mlsanity.com/paper/1606.01164/) (2016)
- [Generalisation error in learning with random features and the hidden manifold model](https://mlsanity.com/paper/2002.09339/) (2020)
- [Large Associative Memory Problem in Neurobiology and Machine Learning](https://mlsanity.com/paper/2008.06996/) (2020)
- [Dense Associative Memory is Robust to Adversarial Inputs](https://mlsanity.com/paper/1701.00939/) (2017)
- [Dreaming neural networks: forgetting spurious memories and reinforcing pure ones](https://mlsanity.com/paper/1810.12217/) (2018)
- [Storage and Learning phase transitions in the Random-Features Hopfield Model](https://mlsanity.com/paper/2303.16880/) (2023)
- [Random Features Hopfield Networks generalize retrieval to previously unseen examples](https://mlsanity.com/paper/2407.05658/) (2024)
- [Dreaming neural networks: rigorous results](https://mlsanity.com/paper/1812.09077/) (2018)
- [Robust exponential memory in Hopfield networks](https://mlsanity.com/paper/1411.4625/) (2014)
- [Hebbian Learning from First Principles](https://mlsanity.com/paper/2401.07110/) (2024)
- [Supervised and Unsupervised protocols for hetero-associative neural networks](https://mlsanity.com/paper/2505.18796/) (2025)
- [Statistical Mechanics of Learning via Reverberation in Bidirectional Associative Memories](https://mlsanity.com/paper/2307.08365/) (2023)
- [A Federated Many-to-One Hopfield model for associative Neural Networks](https://mlsanity.com/paper/2603.19902/) (2026)
- [Finite-size scaling of hetero-associative retrieval in continuous-signal-driven Ising spin systems](https://mlsanity.com/paper/2605.14059/) (2026)

## Abstract (arXiv)

Exponential Hopfield networks store a number of patterns that grows exponentially with the number of neurons, and in their classical formulation they are auto-associative: they complete a corrupted copy of a memory into the memory itself. Many of the tasks one wants such a network to perform are instead hetero-associative, mapping a cue to a different target. We introduce and analyse an exponential neural network of L layers of N binary neurons, each layer carrying its own dataset, whose energy is an exponential of the product of the per-layer Mattis overlaps, so that it is minimised precisely when every layer retrieves the pattern of the same index; the stored association must be a surjective function of the cue, and we show why nothing else can be stored at all. A cavity/signal-to-noise analysis, made exact at leading order by a large-deviation evaluation of the noise, shows that the aligned hetero-associative state is a fixed point of the zero-temperature dynamics up to a number of stored patterns Pc~ eNρL, exponential in the layer size, with an explicit rate ρL that grows like Llog 2; enlarging the basins of attraction lowers the rate but never destroys its exponential character. Comparing the theory with structured data we find that the exponential capacity and the predicted basins survive correlated, many-to-one patterns: the network is a near-perfect content-addressable memory. The same closed forms describe, without refitting, a synthetic manifold, real T-cell-receptor/epitope triples and natural-language intent data, so the mechanism is domain-universal. Generalisation to unseen cues, though significantly above chance, stays below memorisation, and it is the geometry of the encoding, rather than the data domain, that sets how far above chance it reaches. In this family, exponential storage and strong generalisation are distinct capabilities.