# Guiding Visual Attention in Deep Convolutional Neural Networks Based on Human Eye Movements

arXiv: https://arxiv.org/abs/2206.10587 · HTML: https://mlsanity.com/paper/2206.10587/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 86 references that are in the map.
- First posted on arXiv 21 Jun 2022.

## Builds on (most cited first, 14 of 14)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Recurrent Models of Visual Attention](https://mlsanity.com/paper/1406.6247/) (2014)
- [Generalisation in humans and deep neural networks](https://mlsanity.com/paper/1808.08750/) (2018)
- [Generic decoding of seen and imagined objects using hierarchical visual features](https://mlsanity.com/paper/1510.06479/) (2015)
- [Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Brain's Ventral Visual Pathway](https://mlsanity.com/paper/1411.6422/) (2014)
- [Recurrence is required to capture the representational dynamics of the human visual system](https://mlsanity.com/paper/1903.05946/) (2019)
- [Deep Neural Networks Rival the Representation of Primate IT Cortex for Core Visual Object Recognition](https://mlsanity.com/paper/1406.3284/) (2014)
- [Why are Saliency Maps Noisy? Cause of and Solution to Noisy Saliency Maps](https://mlsanity.com/paper/1902.04893/) (2019)
- [Recurrent computations for visual pattern completion](https://mlsanity.com/paper/1706.02240/) (2017)
- [Deep Networks Can Resemble Human Feed-forward Vision in Invariant Object Recognition](https://mlsanity.com/paper/1508.03929/) (2015)
- [Comparing object recognition in humans and deep convolutional neural networks -- An eye tracking study](https://mlsanity.com/paper/2108.00107/) (2021)
- [Deep Learning for Cognitive Neuroscience](https://mlsanity.com/paper/1903.01458/) (2019)

## Built on by (most cited first, 3 of 3)

- [Modeling biological face recognition with deep convolutional neural networks](https://mlsanity.com/paper/2208.06681/) (2022)
- [Perceptual Piercing: Human Visual Cue-based Object Detection in Low Visibility Conditions](https://mlsanity.com/paper/2410.01225/) (2024)
- [From Fog to Failure: The Unintended Consequences of Dehazing on Object Detection in Clear Images](https://mlsanity.com/paper/2502.02027/) (2025)

## Abstract (arXiv)

Deep Convolutional Neural Networks (DCNNs) were originally inspired by principles of biological vision, have evolved into best current computational models of object recognition, and consequently indicate strong architectural and functional parallelism with the ventral visual pathway throughout comparisons with neuroimaging and neural time series data. As recent advances in deep learning seem to decrease this similarity, computational neuroscience is challenged to reverse-engineer the biological plausibility to obtain useful models. While previous studies have shown that biologically inspired architectures are able to amplify the human-likeness of the models, in this study, we investigate a purely data-driven approach. We use human eye tracking data to directly modify training examples and thereby guide the models' visual attention during object recognition in natural images either towards or away from the focus of human fixations. We compare and validate different manipulation types (i.e., standard, human-like, and non-human-like attention) through GradCAM saliency maps against human participant eye tracking data. Our results demonstrate that the proposed guided focus manipulation works as intended in the negative direction and non-human-like models focus on significantly dissimilar image parts compared to humans. The observed effects were highly category-specific, enhanced by animacy and face presence, developed only after feedforward processing was completed, and indicated a strong influence on face detection. With this approach, however, no significantly increased human-likeness was found. Possible applications of overt visual attention in DCNNs and further implications for theories of face detection are discussed.