# Learning Attractor Dynamics for Generative Memory

arXiv: https://arxiv.org/abs/1811.09556 · HTML: https://mlsanity.com/paper/1811.09556/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Nov 2018.

## Built on by (most cited first, 6 of 6)

- [Larimar: Large Language Models with Episodic Memory Control](https://mlsanity.com/paper/2403.11901/) (2024)
- [Attention Approximates Sparse Distributed Memory](https://mlsanity.com/paper/2111.05498/) (2021)
- [BayesPCN: A Continually Learnable Predictive Coding Associative Memory](https://mlsanity.com/paper/2205.09930/) (2022)
- [On the Relationship Between Variational Inference and Auto-Associative Memory](https://mlsanity.com/paper/2210.08013/) (2022)
- [Kanerva++: extending The Kanerva Machine with differentiable, locally block allocated latent memory](https://mlsanity.com/paper/2103.03905/) (2021)
- [Meta-Learning with Variational Bayes](https://mlsanity.com/paper/2103.02265/) (2021)

## Abstract (arXiv)

A central challenge faced by memory systems is the robust retrieval of a stored pattern in the presence of interference due to other stored patterns and noise. A theoretically well-founded solution to robust retrieval is given by attractor dynamics, which iteratively clean up patterns during recall. However, incorporating attractor dynamics into modern deep learning systems poses difficulties: attractor basins are characterised by vanishing gradients, which are known to make training neural networks difficult. In this work, we avoid the vanishing gradient problem by training a generative distributed memory without simulating the attractor dynamics. Based on the idea of memory writing as inference, as proposed in the Kanerva Machine, we show that a likelihood-based Lyapunov function emerges from maximising the variational lower-bound of a generative memory. Experiments shows it converges to correct patterns upon iterative retrieval and achieves competitive performance as both a memory model and a generative model.