# Kanerva++: extending The Kanerva Machine with differentiable, locally block allocated latent memory

arXiv: https://arxiv.org/abs/2103.03905 · HTML: https://mlsanity.com/paper/2103.03905/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 65 references that are in the map.
- First posted on arXiv 20 Feb 2021.

## Builds on (most cited first, 30 of 42)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [Variational Graph Auto-Encoders](https://mlsanity.com/paper/1611.07308/) (2016)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [Neural Turing Machines](https://mlsanity.com/paper/1410.5401/) (2014)
- [Improving Variational Inference with Inverse Autoregressive Flow](https://mlsanity.com/paper/1606.04934/) (2016)
- [Large Batch Training of Convolutional Networks](https://mlsanity.com/paper/1708.03888/) (2017)
- [Importance Weighted Autoencoders](https://mlsanity.com/paper/1509.00519/) (2015)
- [PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications](https://mlsanity.com/paper/1701.05517/) (2017)
- [Memory Networks](https://mlsanity.com/paper/1410.3916/) (2014)
- [A note on the evaluation of generative models](https://mlsanity.com/paper/1511.01844/) (2015)
- [Visual Reinforcement Learning with Imagined Goals](https://mlsanity.com/paper/1807.04742/) (2018)
- [VAE with a VampPrior](https://mlsanity.com/paper/1705.07120/) (2017)
- [Dense Associative Memory for Pattern Recognition](https://mlsanity.com/paper/1606.01164/) (2016)
- [Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates](https://mlsanity.com/paper/1708.07120/) (2017)
- [DRAW: A Recurrent Neural Network For Image Generation](https://mlsanity.com/paper/1502.04623/) (2015)
- [Variational Lossy Autoencoder](https://mlsanity.com/paper/1611.02731/) (2016)
- [Latent Space Oddity: on the Curvature of Deep Generative Models](https://mlsanity.com/paper/1710.11379/) (2017)
- [Neural Episodic Control](https://mlsanity.com/paper/1703.01988/) (2017)
- [PixelVAE: A Latent Variable Model for Natural Images](https://mlsanity.com/paper/1611.05013/) (2016)
- [Z-Forcing: Training Stochastic Recurrent Networks](https://mlsanity.com/paper/1711.05411/) (2017)
- [Evolving Normalization-Activation Layers](https://mlsanity.com/paper/2004.02967/) (2020)
- [Nonparametric Variational Auto-encoders for Hierarchical Representation Learning](https://mlsanity.com/paper/1703.07027/) (2017)

## Built on by (most cited first, 3 of 3)

- [Larimar: Large Language Models with Episodic Memory Control](https://mlsanity.com/paper/2403.11901/) (2024)
- [On the Relationship Between Variational Inference and Auto-Associative Memory](https://mlsanity.com/paper/2210.08013/) (2022)
- [Bootstrap Your Own Variance](https://mlsanity.com/paper/2312.03213/) (2023)

## Abstract (arXiv)

Episodic and semantic memory are critical components of the human memory model. The theory of complementary learning systems (McClelland et al., 1995) suggests that the compressed representation produced by a serial event (episodic memory) is later restructured to build a more generalized form of reusable knowledge (semantic memory). In this work we develop a new principled Bayesian memory allocation scheme that bridges the gap between episodic and semantic memory via a hierarchical latent variable model. We take inspiration from traditional heap allocation and extend the idea of locally contiguous memory to the Kanerva Machine, enabling a novel differentiable block allocated latent memory. In contrast to the Kanerva Machine, we simplify the process of memory writing by treating it as a fully feed forward deterministic process, relying on the stochasticity of the read key distribution to disperse information within the memory. We demonstrate that this allocation scheme improves performance in memory conditional image generation, resulting in new state-of-the-art conditional likelihood values on binarized MNIST (<=41.58 nats/image) , binarized Omniglot (<=66.24 nats/image), as well as presenting competitive performance on CIFAR10, DMLab Mazes, Celeb-A and ImageNet32x32.