# Saliency-based Sequential Image Attention with Multiset Prediction

arXiv: https://arxiv.org/abs/1711.05165 · HTML: https://mlsanity.com/paper/1711.05165/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Nov 2017.

## Built on by (most cited first, 2 of 2)

- [Improving saliency models' predictions of the next fixation with humans' intrinsic cost of gaze shifts](https://mlsanity.com/paper/2207.04250/) (2022)
- [Hard Attention Control By Mutual Information Maximization](https://mlsanity.com/paper/2103.06371/) (2021)

## Abstract (arXiv)

Humans process visual scenes selectively and sequentially using attention. Central to models of human visual attention is the saliency map. We propose a hierarchical visual architecture that operates on a saliency map and uses a novel attention mechanism to sequentially focus on salient regions and take additional glimpses within those regions. The architecture is motivated by human visual attention, and is used for multi-label image classification on a novel multiset task, demonstrating that it achieves high precision and recall while localizing objects with its attention. Unlike conventional multi-label image classification models, the model supports multiset prediction due to a reinforcement-learning based training process that allows for arbitrary label permutation and multiple instances per label.