# Passive Attention in Artificial Neural Networks Predicts Human Visual Selectivity

arXiv: https://arxiv.org/abs/2107.07013 · HTML: https://mlsanity.com/paper/2107.07013/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 56 references that are in the map.
- First posted on arXiv 14 Jul 2021.

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [Striving for Simplicity: The All Convolutional Net](https://mlsanity.com/paper/1412.6806/) (2014)
- [Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networks](https://mlsanity.com/paper/1910.01279/) (2019)
- [How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers](https://mlsanity.com/paper/2106.10270/) (2021)
- [Recurrent Models of Visual Attention](https://mlsanity.com/paper/1406.6247/) (2014)
- [Not Just a Black Box: Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1605.01713/) (2016)
- [Towards Fairness in Visual Recognition: Effective Strategies for Bias Mitigation](https://mlsanity.com/paper/1911.11834/) (2019)
- [REVISE: A Tool for Measuring and Mitigating Bias in Visual Datasets](https://mlsanity.com/paper/2004.07999/) (2020)
- [Human Attention in Visual Question Answering: Do Humans and Deep Networks Look at the Same Regions?](https://mlsanity.com/paper/1606.03556/) (2016)
- [Attention Branch Network: Learning of Attention Mechanism for Visual Explanation](https://mlsanity.com/paper/1812.10025/) (2018)
- [Brain-Like Object Recognition with High-Performing Shallow Recurrent ANNs](https://mlsanity.com/paper/1909.06161/) (2019)
- [Learn To Pay Attention](https://mlsanity.com/paper/1804.02391/) (2018)
- [CAMERAS: Enhanced Resolution And Sanity preserving Class Activation Mapping for image saliency](https://mlsanity.com/paper/2106.10649/) (2021)
- [Emergent Properties of Foveated Perceptual Systems](https://mlsanity.com/paper/2006.07991/) (2020)

## Built on by (most cited first, 10 of 10)

- [Getting aligned on representational alignment](https://mlsanity.com/paper/2310.13018/) (2023)
- [Harmonizing the object recognition strategies of deep neural networks with humans](https://mlsanity.com/paper/2211.04533/) (2022)
- [Words are all you need? Language as an approximation for human similarity judgments](https://mlsanity.com/paper/2206.04105/) (2022)
- [Adversarial alignment: Breaking the trade-off between the strength of an attack and its relevance to human perception](https://mlsanity.com/paper/2306.03229/) (2023)
- [Abstraction Alignment: Comparing Model-Learned and Human-Encoded Conceptual Relationships](https://mlsanity.com/paper/2407.12543/) (2024)
- [Extrapolation Frameworks in Cognitive Psychology Suitable for Study of Image Classification Models](https://mlsanity.com/paper/2112.03411/) (2021)
- [Neural Insights for Digital Marketing Content Design](https://mlsanity.com/paper/2302.01416/) (2023)
- [Attention Alignment Between Humans and Vision-Language Models](https://mlsanity.com/paper/2606.17410/) (2026)
- [Connecting Concept Convexity and Human-Machine Alignment in Deep Neural Networks](https://mlsanity.com/paper/2409.06362/) (2024)
- [Grad-CAM for Vision Transformers: A Systematic Taxonomy and Audit of Methodological Ambiguity in Explainable AI](https://mlsanity.com/paper/2608.05258/) (2026)

## Abstract (arXiv)

Developments in machine learning interpretability techniques over the past decade have provided new tools to observe the image regions that are most informative for classification and localization in artificial neural networks (ANNs). Are the same regions similarly informative to human observers? Using data from 79 new experiments and 7,810 participants, we show that passive attention techniques reveal a significant overlap with human visual selectivity estimates derived from 6 distinct behavioral tasks including visual discrimination, spatial localization, recognizability, free-viewing, cued-object search, and saliency search fixations. We find that input visualizations derived from relatively simple ANN architectures probed using guided backpropagation methods are the best predictors of a shared component in the joint variability of the human measures. We validate these correlational results with causal manipulations using recognition experiments. We show that images masked with ANN attention maps were easier for humans to classify than control masks in a speeded recognition experiment. Similarly, we find that recognition performance in the same ANN models was likewise influenced by masking input images using human visual selectivity maps. This work contributes a new approach to evaluating the biological and psychological validity of leading ANNs as models of human vision: by examining their similarities and differences in terms of their visual selectivity to the information contained in images.