# Overparameterized Neural Networks Implement Associative Memory

arXiv: https://arxiv.org/abs/1909.12362 · HTML: https://mlsanity.com/paper/1909.12362/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Sep 2019.

## Built on by (most cited first, 30 of 45)

- [Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation](https://mlsanity.com/paper/2105.14368/) (2021)
- [Universal Hopfield Networks: A General Framework for Single-Shot Associative Memory Models](https://mlsanity.com/paper/2202.04557/) (2022)
- [Predictive Coding: Towards a Future of Deep Learning beyond Backpropagation?](https://mlsanity.com/paper/2202.09467/) (2022)
- [Associative Memories via Predictive Coding](https://mlsanity.com/paper/2109.08063/) (2021)
- [Key-value memory in the brain](https://mlsanity.com/paper/2501.02950/) (2025)
- [High-dimensional Asymptotics of Denoising Autoencoders](https://mlsanity.com/paper/2305.11041/) (2023)
- [Honest-but-Curious Nets: Sensitive Attributes of Private Inputs Can Be Secretly Coded into the Classifiers' Outputs](https://mlsanity.com/paper/2105.12049/) (2021)
- [DataMUX: Data Multiplexing for Neural Networks](https://mlsanity.com/paper/2202.09318/) (2022)
- [Simple, Fast, and Flexible Framework for Matrix Completion with Infinite Width Neural Networks](https://mlsanity.com/paper/2108.00131/) (2021)
- [Dynamic Loss-Based Sample Reweighting for Improved Large Language Model Pretraining](https://mlsanity.com/paper/2502.06733/) (2025)
- [Do LLMs dream of elephants (when told not to)? Latent concept association and associative memory in transformers](https://mlsanity.com/paper/2406.18400/) (2024)
- [Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?](https://mlsanity.com/paper/2504.12883/) (2025)
- [Content Addressable Memory Without Catastrophic Forgetting by Heteroassociation with a Fixed Scaffold](https://mlsanity.com/paper/2202.00159/) (2022)
- [Non-Parametric Representation Learning with Kernels](https://mlsanity.com/paper/2309.02028/) (2023)
- [Generalization of Diffusion Models Arises with a Balanced Representation Space](https://mlsanity.com/paper/2512.20963/) (2025)
- [Navigating the Latent Space Dynamics of Neural Models](https://mlsanity.com/paper/2505.22785/) (2025)
- [RNNs of RNNs: Recursive Construction of Stable Assemblies of Recurrent Neural Networks](https://mlsanity.com/paper/2106.08928/) (2021)
- [What do larger image classifiers memorise?](https://mlsanity.com/paper/2310.05337/) (2023)
- [Trade-offs in Data Memorization via Strong Data Processing Inequalities](https://mlsanity.com/paper/2506.01855/) (2025)
- [When Diffusion Models Memorize: Inductive Biases in Probability Flow of Minimum-Norm Shallow Neural Nets](https://mlsanity.com/paper/2506.19031/) (2025)
- [A Mechanism for Producing Aligned Latent Spaces with Autoencoders](https://mlsanity.com/paper/2106.15456/) (2021)
- [Transfer Learning with Kernel Methods](https://mlsanity.com/paper/2211.00227/) (2022)
- [Beyond Surprise: Improving Exploration Through Surprise Novelty](https://mlsanity.com/paper/2308.04836/) (2023)
- [Bridging Associative Memory and Probabilistic Modeling](https://mlsanity.com/paper/2402.10202/) (2024)
- [On the Relationship Between Variational Inference and Auto-Associative Memory](https://mlsanity.com/paper/2210.08013/) (2022)
- [Neuro-Causal Factor Analysis](https://mlsanity.com/paper/2305.19802/) (2023)
- [Discrete Diffusion Models Exploit Asymmetry to Solve Lookahead Planning Tasks](https://mlsanity.com/paper/2602.19980/) (2026)
- [Fundamental Limits of Two-layer Autoencoders, and Achieving Them with Gradient Methods](https://mlsanity.com/paper/2212.13468/) (2022)
- [PruMUX: Augmenting Data Multiplexing with Model Compression](https://mlsanity.com/paper/2305.14706/) (2023)
- [Rethinking the Relationship between Recurrent and Non-Recurrent Neural Networks: A Study in Sparsity](https://mlsanity.com/paper/2404.00880/) (2024)

## Abstract (arXiv)

Identifying computational mechanisms for memorization and retrieval of data is a long-standing problem at the intersection of machine learning and neuroscience. Our main finding is that standard overparameterized deep neural networks trained using standard optimization methods implement such a mechanism for real-valued data. Empirically, we show that: (1) overparameterized autoencoders store training samples as attractors, and thus, iterating the learned map leads to sample recovery; (2) the same mechanism allows for encoding sequences of examples, and serves as an even more efficient mechanism for memory than autoencoding. Theoretically, we prove that when trained on a single example, autoencoders store the example as an attractor. Lastly, by treating a sequence encoder as a composition of maps, we prove that sequence encoding provides a more efficient mechanism for memory than autoencoding.