# Associative Memory in Iterated Overparameterized Sigmoid Autoencoders

arXiv: https://arxiv.org/abs/2006.16540 · HTML: https://mlsanity.com/paper/2006.16540/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Jun 2020.

## Built on by (most cited first, 9 of 9)

- [Birth of a Transformer: A Memory Viewpoint](https://mlsanity.com/paper/2306.00802/) (2023)
- [Universal Hopfield Networks: A General Framework for Single-Shot Associative Memory Models](https://mlsanity.com/paper/2202.04557/) (2022)
- [Do LLMs dream of elephants (when told not to)? Latent concept association and associative memory in transformers](https://mlsanity.com/paper/2406.18400/) (2024)
- [Navigating the Latent Space Dynamics of Neural Models](https://mlsanity.com/paper/2505.22785/) (2025)
- [When Diffusion Models Memorize: Inductive Biases in Probability Flow of Minimum-Norm Shallow Neural Nets](https://mlsanity.com/paper/2506.19031/) (2025)
- [Autoencoder Attractors for Uncertainty Estimation](https://mlsanity.com/paper/2204.00382/) (2022)
- [How Much Training Data is Memorized in Overparameterized Autoencoders? An Inverse Problem Perspective on Memorization Evaluation](https://mlsanity.com/paper/2310.02897/) (2023)
- [Optimal Representation Size: High-Dimensional Analysis of Pretraining and Linear Probing](https://mlsanity.com/paper/2605.20105/) (2026)
- [Second-order Symmetric Non-negative Latent Factor Analysis](https://mlsanity.com/paper/2203.02088/) (2022)

## Abstract (arXiv)

Recent work showed that overparameterized autoencoders can be trained to implement associative memory via iterative maps, when the trained input-output Jacobian of the network has all of its eigenvalue norms strictly below one. Here, we theoretically analyze this phenomenon for sigmoid networks by leveraging recent developments in deep learning theory, especially the correspondence between training neural networks in the infinite-width limit and performing kernel regression with the Neural Tangent Kernel (NTK). We find that overparameterized sigmoid autoencoders can have attractors in the NTK limit for both training with a single example and multiple examples under certain conditions. In particular, for multiple training examples, we find that the norm of the largest Jacobian eigenvalue drops below one with increasing input norm, leading to associative memory.