# Learning what and where to attend

arXiv: https://arxiv.org/abs/1805.08819 · HTML: https://mlsanity.com/paper/1805.08819/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 May 2018.

## Built on by (most cited first, 8 of 8)

- [Visual Test-time Scaling for GUI Agent Grounding](https://mlsanity.com/paper/2505.00684/) (2025)
- [Understanding the computational demands underlying visual reasoning](https://mlsanity.com/paper/2108.03603/) (2021)
- [Individual Explanations in Machine Learning Models: A Survey for Practitioners](https://mlsanity.com/paper/2104.04144/) (2021)
- [On the Relationship Between Interpretability and Explainability in Machine Learning](https://mlsanity.com/paper/2311.11491/) (2023)
- [ShuffleBlock: Shuffle to Regularize Deep Convolutional Neural Networks](https://mlsanity.com/paper/2106.09358/) (2021)
- [PhD Thesis: Exploring the role of (self-)attention in cognitive and computer vision architecture](https://mlsanity.com/paper/2306.14650/) (2023)
- [Diagnosing and exploiting the computational demands of videos games for deep reinforcement learning](https://mlsanity.com/paper/2309.13181/) (2023)
- [3A-YOLO: New Real-Time Object Detectors with Triple Discriminative Awareness and Coordinated Representations](https://mlsanity.com/paper/2412.07168/) (2024)

## Abstract (arXiv)

Most recent gains in visual recognition have originated from the inclusion of attention mechanisms in deep convolutional networks (DCNs). Because these networks are optimized for object recognition, they learn where to attend using only a weak form of supervision derived from image class labels. Here, we demonstrate the benefit of using stronger supervisory signals by teaching DCNs to attend to image regions that humans deem important for object recognition. We first describe a large-scale online experiment (ClickMe) used to supplement ImageNet with nearly half a million human-derived "top-down" attention maps. Using human psychophysics, we confirm that the identified top-down features from ClickMe are more diagnostic than "bottom-up" saliency features for rapid image categorization. As a proof of concept, we extend a state-of-the-art attention network and demonstrate that adding ClickMe supervision significantly improves its accuracy and yields visual features that are more interpretable and more similar to those used by human observers.