mlsanityData to 7 Oct 2026

Overparameterized Neural Networks Implement Associative Memory

arXiv 1909.12362 ↗First posted 26 Sep 2019cs.LG stat.MLcited by 45

Not on the map. Why?

Abstract (arXiv)

Identifying computational mechanisms for memorization and retrieval of data is a long-standing problem at the intersection of machine learning and neuroscience. Our main finding is that standard overparameterized deep neural networks trained using standard optimization methods implement such a mechanism for real-valued data. Empirically, we show that: (1) overparameterized autoencoders store training samples as attractors, and thus, iterating the learned map leads to sample recovery; (2) the same mechanism allows for encoding sequences of examples, and serves as an even more efficient mechanism for memory than autoencoding. Theoretically, we prove that when trained on a single example, autoencoders store the example as an attractor. Lastly, by treating a sequence encoder as a composition of maps, we prove that sequence encoding provides a more efficient mechanism for memory than autoencoding.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). First posted on arXiv 26 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 10; 2023: 10; 2024: 6; 2025: 9; 2026: 3.

45 papers.

Sort:

By area:

  1. 2021Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolationcited by 130
  2. 2022Universal Hopfield Networks: A General Framework for Single-Shot Associative Memory Modelscited by 52
  3. 2022Predictive Coding: Towards a Future of Deep Learning beyond Backpropagation?cited by 45
  4. 2021Associative Memories via Predictive Codingcited by 42
  5. 2025Key-value memory in the braincited by 22
  6. 2023High-dimensional Asymptotics of Denoising Autoencoderscited by 19
  7. 2021Honest-but-Curious Nets: Sensitive Attributes of Private Inputs Can Be Secretly Coded into the Classifiers' Outputscited by 18
  8. 2022DataMUX: Data Multiplexing for Neural Networkscited by 18
  9. 2021Simple, Fast, and Flexible Framework for Matrix Completion with Infinite Width Neural Networkscited by 17
  10. 2025Dynamic Loss-Based Sample Reweighting for Improved Large Language Model Pretrainingcited by 16
  11. 2024Do LLMs dream of elephants (when told not to)? Latent concept association and associative memory in transformerscited by 14
  12. 2022Content Addressable Memory Without Catastrophic Forgetting by Heteroassociation with a Fixed Scaffoldcited by 11
33 more
  1. 2023Non-Parametric Representation Learning with Kernelscited by 11
  2. 2025Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?cited by 10
  3. 2025Generalization of Diffusion Models Arises with a Balanced Representation Spacecited by 10
  4. 2021RNNs of RNNs: Recursive Construction of Stable Assemblies of Recurrent Neural Networkscited by 7
  5. 2025Navigating the Latent Space Dynamics of Neural Modelscited by 7
  6. 2023What do larger image classifiers memorise?cited by 6
  7. 2025When Diffusion Models Memorize: Inductive Biases in Probability Flow of Minimum-Norm Shallow Neural Netscited by 6
  8. 2021A Mechanism for Producing Aligned Latent Spaces with Autoencoderscited by 5
  9. 2022Transfer Learning with Kernel Methodscited by 5
  10. 2023Beyond Surprise: Improving Exploration Through Surprise Noveltycited by 5
  11. 2024Bridging Associative Memory and Probabilistic Modelingcited by 5
  12. 2025Trade-offs in Data Memorization via Strong Data Processing Inequalitiescited by 5
  13. 2022On the Relationship Between Variational Inference and Auto-Associative Memorycited by 4
  14. 2023Neuro-Causal Factor Analysiscited by 4
  15. 2026Discrete Diffusion Models Exploit Asymmetry to Solve Lookahead Planning Taskscited by 4
  16. 2022Fundamental Limits of Two-layer Autoencoders, and Achieving Them with Gradient Methodscited by 3
  17. 2023PruMUX: Augmenting Data Multiplexing with Model Compressioncited by 3
  18. 2024Rethinking the Relationship between Recurrent and Non-Recurrent Neural Networks: A Study in Sparsitycited by 3
  19. 2024Self-attention as an attractor network: transient memories without backpropagationcited by 3
  20. 2021Geometry and Generalization: Eigenvalues as predictors of where a network will fail to generalizecited by 2
  21. 2022Autoencoder Attractors for Uncertainty Estimationcited by 2
  22. 2023How Much Training Data is Memorized in Overparameterized Autoencoders? An Inverse Problem Perspective on Memorization Evaluationcited by 2
  23. 2024Empirical Capacity Model for Self-Attention Neural Networkscited by 1
  24. 2025How Does Overparameterization Affect Machine Unlearning of Deep Neural Networks?cited by 1
  25. 2022Matryoshka: Stealing Functionality of Private ML Data by Hiding Models in Modelcited by 0
  26. 2022Bird-Eye Transformers for Text Generation Modelscited by 0
  27. 2023Fundamental Limits of Deep Learning-Based Binary Classifiers Trained with Hinge Losscited by 0
  28. 2023Semiotics Networks Representing Perceptual Inferencecited by 0
  29. 2023Unified machine learning tasks and datasets for enhancing renewable energycited by 0
  30. 2024Evaluation of Predictive Reliability to Foster Trust in Artificial Intelligence. A case study in Multiple Sclerosiscited by 0
  31. 2025A Non-Adversarial Approach to Idempotent Generative Modellingcited by 0
  32. 2026Optimal Representation Size: High-Dimensional Analysis of Pretraining and Linear Probingcited by 0
  33. 2026Self-Reconstruction Dynamics for Autoencoder Reconstruction Refinementcited by 0