# Generative Temporal Models with Memory

arXiv: https://arxiv.org/abs/1702.04649 · HTML: https://mlsanity.com/paper/1702.04649/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Feb 2017.

## Built on by (most cited first, 6 of 6)

- [Clockwork Variational Autoencoders](https://mlsanity.com/paper/2102.09532/) (2021)
- [Memory and attention in deep learning](https://mlsanity.com/paper/2107.01390/) (2021)
- [BAM: Bayes with Adaptive Memory](https://mlsanity.com/paper/2202.02405/) (2022)
- [A Review of Symbolic, Subsymbolic and Hybrid Methods for Sequential Decision Making](https://mlsanity.com/paper/2304.10590/) (2023)
- [Imitation Learning of Factored Multi-agent Reactive Models](https://mlsanity.com/paper/1903.04714/) (2019)
- [A Biologically-Inspired Dual Stream World Model](https://mlsanity.com/paper/2209.08035/) (2022)

## Abstract (arXiv)

We consider the general problem of modeling temporal data with long-range dependencies, wherein new observations are fully or partially predictable based on temporally-distant, past observations. A sufficiently powerful temporal model should separate predictable elements of the sequence from unpredictable elements, express uncertainty about those unpredictable elements, and rapidly identify novel elements that may help to predict the future. To create such models, we introduce Generative Temporal Models augmented with external memory systems. They are developed within the variational inference framework, which provides both a practical training methodology and methods to gain insight into the models' operation. We show, on a range of problems with sparse, long-term temporal dependencies, that these models store information from early in a sequence, and reuse this stored information efficiently. This allows them to perform substantially better than existing models based on well-known recurrent neural networks, like LSTMs.