# A Probabilistic Hard Attention Model For Sequentially Observed Scenes arXiv: https://arxiv.org/abs/2111.07534 ยท HTML: https://mlsanity.com/paper/2111.07534/ ## Key facts - Cited by 12 papers in the map (data through 14 Oct 2026). - Builds on 37 of its 69 references that are in the map. - First posted on arXiv 15 Nov 2021. ## Builds on (most cited first, 30 of 37) - [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014) - [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015) - [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015) - [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016) - [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013) - [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016) - [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018) - [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015) - [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016) - [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015) - [Attention-based Deep Multiple Instance Learning](https://mlsanity.com/paper/1802.04712/) (2018) - [Improving Variational Inference with Inverse Autoregressive Flow](https://mlsanity.com/paper/1606.04934/) (2016) - [Neural Spline Flows](https://mlsanity.com/paper/1906.04032/) (2019) - [Generative Image Inpainting with Contextual Attention](https://mlsanity.com/paper/1801.07892/) (2018) - [Conditional Neural Processes](https://mlsanity.com/paper/1807.01613/) (2018) - [Recurrent Models of Visual Attention](https://mlsanity.com/paper/1406.6247/) (2014) - [A Structured Self-attentive Sentence Embedding](https://mlsanity.com/paper/1703.03130/) (2017) - [Neural Processes](https://mlsanity.com/paper/1807.01622/) (2018) - [Attend, Infer, Repeat: Fast Scene Understanding with Generative Models](https://mlsanity.com/paper/1603.08575/) (2016) - [DRAW: A Recurrent Neural Network For Image Generation](https://mlsanity.com/paper/1502.04623/) (2015) - [Pluralistic Image Completion](https://mlsanity.com/paper/1903.04227/) (2019) - [EDDI: Efficient Dynamic Discovery of High-Value Information with Partial VAE](https://mlsanity.com/paper/1809.11142/) (2018) - [Differentiable Patch Selection for Image Recognition](https://mlsanity.com/paper/2104.03059/) (2021) - [Very Long Natural Scenery Image Prediction by Outpainting](https://mlsanity.com/paper/1912.12688/) (2019) - [Learning When and Where to Zoom with Deep Reinforcement Learning](https://mlsanity.com/paper/2003.00425/) (2020) - [Saccader: Improving Accuracy of Hard Attention Models for Vision](https://mlsanity.com/paper/1908.07644/) (2019) - [Processing Megapixel Images with Deep Attention-Sampling Models](https://mlsanity.com/paper/1905.03711/) (2019) - [Dynamic Computational Time for Visual Attention](https://mlsanity.com/paper/1703.10332/) (2017) - [Hard-Attention for Scalable Image Classification](https://mlsanity.com/paper/2102.10212/) (2021) - [PiiGAN: Generative Adversarial Networks for Pluralistic Image Inpainting](https://mlsanity.com/paper/1912.01834/) (2019) ## Built on by (most cited first, 12 of 12) - [Variational Information Pursuit for Interpretable Predictions](https://mlsanity.com/paper/2302.02876/) (2023) - [Estimating Conditional Mutual Information for Dynamic Feature Selection](https://mlsanity.com/paper/2306.03301/) (2023) - [Stochastic Amortization: A Unified Approach to Accelerate Feature and Data Attribution](https://mlsanity.com/paper/2401.15866/) (2024) - [A Survey on Active Feature Acquisition Strategies](https://mlsanity.com/paper/2502.11067/) (2025) - [Consistency driven Sequential Transformers Attention Model for Partially Observable Scenes](https://mlsanity.com/paper/2204.00656/) (2022) - [Stochastic Encodings for Active Feature Acquisition](https://mlsanity.com/paper/2508.01957/) (2025) - [AdaGlimpse: Active Visual Exploration with Arbitrary Glimpse Position and Scale](https://mlsanity.com/paper/2404.03482/) (2024) - [Aerial View Localization with Reinforcement Learning: Towards Emulating Search-and-Rescue](https://mlsanity.com/paper/2209.03694/) (2022) - [GliTr: Glimpse Transformers with Spatiotemporal Consistency for Online Action Prediction](https://mlsanity.com/paper/2210.13605/) (2022) - [SCAI: A Spectral data Classification framework with Adaptive Inference for the IoT platform](https://mlsanity.com/paper/2206.12420/) (2022) - [Energy-Efficient Visual Search by Eye Movement and Low-Latency Spiking Neural Network](https://mlsanity.com/paper/2310.06578/) (2023) - [Language-Guided Reinforcement Learning for Hard Attention in Few-Shot Learning](https://mlsanity.com/paper/2310.07800/) (2023) ## Abstract (arXiv) A visual hard attention model actively selects and observes a sequence of subregions in an image to make a prediction. The majority of hard attention models determine the attention-worthy regions by first analyzing a complete image. However, it may be the case that the entire image is not available initially but instead sensed gradually through a series of partial observations. In this paper, we design an efficient hard attention model for classifying such sequentially observed scenes. The presented model never observes an image completely. To select informative regions under partial observability, the model uses Bayesian Optimal Experiment Design. First, it synthesizes the features of the unobserved regions based on the already observed regions. Then, it uses the predicted features to estimate the expected information gain (EIG) attained, should various regions be attended. Finally, the model attends to the actual content on the location where the EIG mentioned above is maximum. The model uses a) a recurrent feature aggregator to maintain a recurrent state, b) a linear classifier to predict the class label, c) a Partial variational autoencoder to predict the features of unobserved regions. We use normalizing flows in Partial VAE to handle multi-modality in the feature-synthesis problem. We train our model using a differentiable objective and test it on five datasets. Our model gains 2-10% higher accuracy than the baseline models when both have seen only a couple of glimpses.