# The Kanerva Machine: A Generative Distributed Memory

arXiv: https://arxiv.org/abs/1804.01756 · HTML: https://mlsanity.com/paper/1804.01756/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Apr 2018.

## Built on by (most cited first, 14 of 14)

- [Continual Learning of Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2404.16789/) (2024)
- [Larimar: Large Language Models with Episodic Memory Control](https://mlsanity.com/paper/2403.11901/) (2024)
- [Attention Approximates Sparse Distributed Memory](https://mlsanity.com/paper/2111.05498/) (2021)
- [Abstraction for Deep Reinforcement Learning](https://mlsanity.com/paper/2202.05839/) (2022)
- [BayesPCN: A Continually Learnable Predictive Coding Associative Memory](https://mlsanity.com/paper/2205.09930/) (2022)
- [Memory-Augmented Transformers: A Systematic Review from Neuroscience Principles to Enhanced Model Architectures](https://mlsanity.com/paper/2508.10824/) (2025)
- [On the Relationship Between Variational Inference and Auto-Associative Memory](https://mlsanity.com/paper/2210.08013/) (2022)
- [Kanerva++: extending The Kanerva Machine with differentiable, locally block allocated latent memory](https://mlsanity.com/paper/2103.03905/) (2021)
- [BAM: Bayes with Adaptive Memory](https://mlsanity.com/paper/2202.02405/) (2022)
- [A learning gap between neuroscience and reinforcement learning](https://mlsanity.com/paper/2104.10995/) (2021)
- [Bootstrap Your Own Variance](https://mlsanity.com/paper/2312.03213/) (2023)
- [Meta-Learning with Variational Bayes](https://mlsanity.com/paper/2103.02265/) (2021)
- [Active perception and disentangled representations allow continual, episodic zero and few-shot learning](https://mlsanity.com/paper/2602.19355/) (2026)
- [Continuous Memory Machines](https://mlsanity.com/paper/2610.07907/) (2026)

## Abstract (arXiv)

We present an end-to-end trained memory system that quickly adapts to new data and generates samples like them. Inspired by Kanerva's sparse distributed memory, it has a robust distributed reading and writing mechanism. The memory is analytically tractable, which enables optimal on-line compression via a Bayesian update-rule. We formulate it as a hierarchical conditional generative model, where memory provides a rich data-dependent prior distribution. Consequently, the top-down memory and bottom-up perception are combined to produce the code representing an observation. Empirically, we demonstrate that the adaptive memory significantly improves generative models trained on both the Omniglot and CIFAR datasets. Compared with the Differentiable Neural Computer (DNC) and its variants, our memory model has greater capacity and is significantly easier to train.