# A model of semantic completion in generative episodic memory

arXiv: https://arxiv.org/abs/2111.13537 · HTML: https://mlsanity.com/paper/2111.13537/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 80 references that are in the map.
- First posted on arXiv 26 Nov 2021.

## Builds on (most cited first, 13 of 13)

- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [Conditional Image Generation with PixelCNN Decoders](https://mlsanity.com/paper/1606.05328/) (2016)
- [Image Transformer](https://mlsanity.com/paper/1802.05751/) (2018)
- [Deep multi-scale video prediction beyond mean square error](https://mlsanity.com/paper/1511.05440/) (2015)
- [Convolutional Neural Networks as a Model of the Visual System: Past, Present, and Future](https://mlsanity.com/paper/2001.07092/) (2020)
- [Predicting Video with VQVAE](https://mlsanity.com/paper/2103.01950/) (2021)
- [Recurrent computations for visual pattern completion](https://mlsanity.com/paper/1706.02240/) (2017)
- [Stacked autoencoders based machine learning for noise reduction and signal reconstruction in geophysical data](https://mlsanity.com/paper/1907.03278/) (2019)
- [A Hippocampus Model for Online One-Shot Storage of Pattern Sequences](https://mlsanity.com/paper/1905.12937/) (2019)

## Built on by (most cited first, 5 of 5)

- [MemoryKT: An Integrative Memory-and-Forgetting Method for Knowledge Tracing](https://mlsanity.com/paper/2508.08122/) (2025)
- [GENESIS: A Generative Model of Episodic-Semantic Interaction](https://mlsanity.com/paper/2510.15828/) (2025)
- [Understanding Transformer-based Vision Models through Inversion](https://mlsanity.com/paper/2412.06534/) (2024)
- [Is Hierarchical Quantization Essential for Optimal Reconstruction?](https://mlsanity.com/paper/2601.22244/) (2026)
- [The Dynamic Gist-Based Memory Model (DGMM): A Memory-Centric Architecture for Artificial Intelligence](https://mlsanity.com/paper/2605.02106/) (2026)

## Abstract (arXiv)

Many different studies have suggested that episodic memory is a generative process, but most computational models adopt a storage view. In this work, we propose a computational model for generative episodic memory. It is based on the central hypothesis that the hippocampus stores and retrieves selected aspects of an episode as a memory trace, which is necessarily incomplete. At recall, the neocortex reasonably fills in the missing information based on general semantic information in a process we call semantic completion. As episodes we use images of digits (MNIST) augmented by different backgrounds representing context. Our model is based on a VQ-VAE which generates a compressed latent representation in form of an index matrix, which still has some spatial resolution. We assume that attention selects some part of the index matrix while others are discarded, this then represents the gist of the episode and is stored as a memory trace. At recall the missing parts are filled in by a PixelCNN, modeling semantic completion, and the completed index matrix is then decoded into a full image by the VQ-VAE. The model is able to complete missing parts of a memory trace in a semantically plausible way up to the point where it can generate plausible images from scratch. Due to the combinatorics in the index matrix, the model generalizes well to images not trained on. Compression as well as semantic completion contribute to a strong reduction in memory requirements and robustness to noise. Finally we also model an episodic memory experiment and can reproduce that semantically congruent contexts are always recalled better than incongruent ones, high attention levels improve memory accuracy in both cases, and contexts that are not remembered correctly are more often remembered semantically congruently than completely wrong.