# Learning Wake-Sleep Recurrent Attention Models

arXiv: https://arxiv.org/abs/1509.06812 · HTML: https://mlsanity.com/paper/1509.06812/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Sep 2015.

## Built on by (most cited first, 9 of 9)

- [Attention Mechanism in Neural Networks: Where it Comes and Where it Goes](https://mlsanity.com/paper/2204.13154/) (2022)
- [Estimating Conditional Mutual Information for Dynamic Feature Selection](https://mlsanity.com/paper/2306.03301/) (2023)
- [A Probabilistic Hard Attention Model For Sequentially Observed Scenes](https://mlsanity.com/paper/2111.07534/) (2021)
- [Inducing and Using Alignments for Transition-based AMR Parsing](https://mlsanity.com/paper/2205.01464/) (2022)
- [Consistency driven Sequential Transformers Attention Model for Partially Observable Scenes](https://mlsanity.com/paper/2204.00656/) (2022)
- [AdaGlimpse: Active Visual Exploration with Arbitrary Glimpse Position and Scale](https://mlsanity.com/paper/2404.03482/) (2024)
- [Uphill Roads to Variational Tightness: Monotonicity and Monte Carlo Objectives](https://mlsanity.com/paper/2201.10989/) (2022)
- [Visual Attention in Imaginative Agents](https://mlsanity.com/paper/2104.00177/) (2021)
- [Language-Guided Reinforcement Learning for Hard Attention in Few-Shot Learning](https://mlsanity.com/paper/2310.07800/) (2023)

## Abstract (arXiv)

Despite their success, convolutional neural networks are computationally expensive because they must examine all image locations. Stochastic attention-based models have been shown to improve computational efficiency at test time, but they remain difficult to train because of intractable posterior inference and high variance in the stochastic gradient estimates. Borrowing techniques from the literature on training deep generative models, we present the Wake-Sleep Recurrent Attention Model, a method for training stochastic attention networks which improves posterior inference and which reduces the variability in the stochastic gradients. We show that our method can greatly speed up the training time for stochastic attention networks in the domains of image classification and caption generation.