# Learning to Generate with Memory

arXiv: https://arxiv.org/abs/1602.07416 · HTML: https://mlsanity.com/paper/1602.07416/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Feb 2016.

## Built on by (most cited first, 6 of 6)

- [Attention, please! A survey of Neural Attention Models in Deep Learning](https://mlsanity.com/paper/2103.16775/) (2021)
- [Graph-level Anomaly Detection via Hierarchical Memory Networks](https://mlsanity.com/paper/2307.00755/) (2023)
- [OLED: One-Class Learned Encoder-Decoder Network with Adversarial Context Masking for Novelty Detection](https://mlsanity.com/paper/2103.14953/) (2021)
- [SimulLR: Simultaneous Lip Reading Transducer with Attention-Guided Adaptive Memory](https://mlsanity.com/paper/2108.13630/) (2021)
- [Conditional Sampling of Variational Autoencoders via Iterated Approximate Ancestral Sampling](https://mlsanity.com/paper/2308.09078/) (2023)
- [NeurInt : Learning to Interpolate through Neural ODEs](https://mlsanity.com/paper/2111.04123/) (2021)

## Abstract (arXiv)

Memory units have been widely used to enrich the capabilities of deep networks on capturing long-term dependencies in reasoning and prediction tasks, but little investigation exists on deep generative models (DGMs) which are good at inferring high-level invariant representations from unlabeled data. This paper presents a deep generative model with a possibly large external memory and an attention mechanism to capture the local detail information that is often lost in the bottom-up abstraction process in representation learning. By adopting a smooth attention model, the whole network is trained end-to-end by optimizing a variational bound of data likelihood via auto-encoding variational Bayesian methods, where an asymmetric recognition network is learnt jointly to infer high-level invariant representations. The asymmetric architecture can reduce the competition between bottom-up invariant feature extraction and top-down generation of instance details. Our experiments on several datasets demonstrate that memory can significantly boost the performance of DGMs and even achieve state-of-the-art results on various tasks, including density estimation, image generation, and missing value imputation.