# Behind the Machine's Gaze: Neural Networks with Biologically-inspired Constraints Exhibit Human-like Visual Attention

arXiv: https://arxiv.org/abs/2204.09093 · HTML: https://mlsanity.com/paper/2204.09093/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 75 references that are in the map.
- First posted on arXiv 19 Apr 2022.

## Builds on (most cited first, 25 of 25)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising](https://mlsanity.com/paper/1608.03981/) (2016)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [Recurrent Models of Visual Attention](https://mlsanity.com/paper/1406.6247/) (2014)
- [Wider or Deeper: Revisiting the ResNet Model for Visual Recognition](https://mlsanity.com/paper/1611.10080/) (2016)
- [Human Attention in Visual Question Answering: Do Humans and Deep Networks Look at the Same Regions?](https://mlsanity.com/paper/1606.03556/) (2016)
- [DeepGaze II: Reading fixations from deep features trained on object recognition](https://mlsanity.com/paper/1610.01563/) (2016)
- [CAT2000: A Large Scale Fixation Dataset for Boosting Saliency Research](https://mlsanity.com/paper/1505.03581/) (2015)
- [Exploring Misclassifications of Robust Neural Networks to Enhance Adversarial Attacks](https://mlsanity.com/paper/2105.10304/) (2021)
- [Foveation-based Mechanisms Alleviate Adversarial Examples](https://mlsanity.com/paper/1511.06292/) (2015)
- [PathGAN: Visual Scanpath Prediction with Generative Adversarial Networks](https://mlsanity.com/paper/1809.00567/) (2018)
- [Biologically Inspired Mechanisms for Adversarial Robustness](https://mlsanity.com/paper/2006.16427/) (2020)
- [State-of-the-Art in Human Scanpath Prediction](https://mlsanity.com/paper/2102.12239/) (2021)
- [Identifying Untrustworthy Predictions in Neural Networks by Geometric Gradient Analysis](https://mlsanity.com/paper/2102.12196/) (2021)
- [Toward Improving the Evaluation of Visual Attention Models: a Crowdsourcing Approach](https://mlsanity.com/paper/2002.04407/) (2020)
- [Gravitational Models Explain Shifts on Human Visual Attention](https://mlsanity.com/paper/2009.06963/) (2020)
- [Region-of-Interest Prioritised Sampling for Constrained Autonomous Exploration Systems](https://mlsanity.com/paper/2107.07186/) (2021)

## Built on by (most cited first, 7 of 7)

- [EyeFormer: Predicting Personalized Scanpaths with Transformer-Guided Reinforcement Learning](https://mlsanity.com/paper/2404.10163/) (2024)
- [LLMind: Bio-inspired Training-free Adaptive Visual Representations for Vision-Language Models](https://mlsanity.com/paper/2603.14882/) (2026)
- [OAT: Object-Level Attention Transformer for Gaze Scanpath Prediction](https://mlsanity.com/paper/2407.13335/) (2024)
- [Contrastive Language-Image Pretrained Models are Zero-Shot Human Scanpath Predictors](https://mlsanity.com/paper/2305.12380/) (2023)
- [FovEx: Human-Inspired Explanations for Vision Transformers and Convolutional Neural Networks](https://mlsanity.com/paper/2408.02123/) (2024)
- [Caption-Driven Explorations: Aligning Image and Text Embeddings through Human-Inspired Foveated Vision](https://mlsanity.com/paper/2408.09948/) (2024)
- [Matched Outcomes, Divergent Gaze: How Foveated MLLMs Search Compared to Humans](https://mlsanity.com/paper/2608.16514/) (2026)

## Abstract (arXiv)

By and large, existing computational models of visual attention tacitly assume perfect vision and full access to the stimulus and thereby deviate from foveated biological vision. Moreover, modeling top-down attention is generally reduced to the integration of semantic features without incorporating the signal of a high-level visual tasks that have been shown to partially guide human attention. We propose the Neural Visual Attention (NeVA) algorithm to generate visual scanpaths in a top-down manner. With our method, we explore the ability of neural networks on which we impose a biologically-inspired foveated vision constraint to generate human-like scanpaths without directly training for this objective. The loss of a neural network performing a downstream visual task (i.e., classification or reconstruction) flexibly provides top-down guidance to the scanpath. Extensive experiments show that our method outperforms state-of-the-art unsupervised human attention models in terms of similarity to human scanpaths. Additionally, the flexibility of the framework allows to quantitatively investigate the role of different tasks in the generated visual behaviors. Finally, we demonstrate the superiority of the approach in a novel experiment that investigates the utility of scanpaths in real-world applications, where imperfect viewing conditions are given.