# Comparing object recognition in humans and deep convolutional neural networks -- An eye tracking study

arXiv: https://arxiv.org/abs/2108.00107 · HTML: https://mlsanity.com/paper/2108.00107/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 62 references that are in the map.
- First posted on arXiv 30 Jul 2021.

## Builds on (most cited first, 20 of 20)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Recognition in Terra Incognita](https://mlsanity.com/paper/1807.04975/) (2018)
- [Understanding the Effective Receptive Field in Deep Convolutional Neural Networks](https://mlsanity.com/paper/1701.04128/) (2017)
- [Generalisation in humans and deep neural networks](https://mlsanity.com/paper/1808.08750/) (2018)
- [Strike (with) a Pose: Neural Networks Are Easily Fooled by Strange Poses of Familiar Objects](https://mlsanity.com/paper/1811.11553/) (2018)
- [A Study and Comparison of Human and Deep Learning Recognition Performance Under Visual Distortions](https://mlsanity.com/paper/1705.02498/) (2017)
- [Deep Neural Networks Rival the Representation of Primate IT Cortex for Core Visual Object Recognition](https://mlsanity.com/paper/1406.3284/) (2014)
- [Comparing deep neural networks against humans: object recognition when the signal gets weaker](https://mlsanity.com/paper/1706.06969/) (2017)
- [Recurrent computations for visual pattern completion](https://mlsanity.com/paper/1706.02240/) (2017)
- [Five Points to Check when Comparing Visual Perception in Humans and Machines](https://mlsanity.com/paper/2004.09406/) (2020)
- [The temporal evolution of the central fixation bias in scene viewing](https://mlsanity.com/paper/1610.05982/) (2016)
- [Seeing eye-to-eye? A comparison of object recognition performance in humans and deep convolutional neural networks under image manipulation](https://mlsanity.com/paper/2007.06294/) (2020)

## Built on by (most cited first, 12 of 12)

- [A Comprehensive Survey of Convolutions in Deep Learning: Applications, Challenges, and Future Trends](https://mlsanity.com/paper/2402.15490/) (2024)
- [Do humans and Convolutional Neural Networks attend to similar areas during scene classification: Effects of task and image type](https://mlsanity.com/paper/2307.13345/) (2023)
- [Temporal DINO: A Self-supervised Video Strategy to Enhance Action Prediction](https://mlsanity.com/paper/2308.04589/) (2023)
- [Guiding Visual Attention in Deep Convolutional Neural Networks Based on Human Eye Movements](https://mlsanity.com/paper/2206.10587/) (2022)
- [Gazing at Rewards: Eye Movements as a Lens into Human and AI Decision-Making in Hybrid Visual Foraging](https://mlsanity.com/paper/2411.09176/) (2024)
- [Symmetry Perception by Deep Networks: Inadequacy of Feed-Forward Architectures and Improvements with Recurrent Connections](https://mlsanity.com/paper/2112.04162/) (2021)
- [Brain informed transfer learning for categorizing construction hazards](https://mlsanity.com/paper/2211.12420/) (2022)
- [Reinforcement Learning-based Mixture of Vision Transformers for Video Violence Recognition](https://mlsanity.com/paper/2310.03108/) (2023)
- [How much data do I need? A case study on medical data](https://mlsanity.com/paper/2311.15331/) (2023)
- [Selecting Interpretability Techniques for Healthcare Machine Learning models](https://mlsanity.com/paper/2406.10213/) (2024)
- [Perceptual Piercing: Human Visual Cue-based Object Detection in Low Visibility Conditions](https://mlsanity.com/paper/2410.01225/) (2024)
- [From Fog to Failure: The Unintended Consequences of Dehazing on Object Detection in Clear Images](https://mlsanity.com/paper/2502.02027/) (2025)

## Abstract (arXiv)

Deep convolutional neural networks (DCNNs) and the ventral visual pathway share vast architectural and functional similarities in visual challenges such as object recognition. Recent insights have demonstrated that both hierarchical cascades can be compared in terms of both exerted behavior and underlying activation. However, these approaches ignore key differences in spatial priorities of information processing. In this proof-of-concept study, we demonstrate a comparison of human observers (N = 45) and three feedforward DCNNs through eye tracking and saliency maps. The results reveal fundamentally different resolutions in both visualization methods that need to be considered for an insightful comparison. Moreover, we provide evidence that a DCNN with biologically plausible receptive field sizes called vNet reveals higher agreement with human viewing behavior as contrasted with a standard ResNet architecture. We find that image-specific factors such as category, animacy, arousal, and valence have a direct link to the agreement of spatial object recognition priorities in humans and DCNNs, while other measures such as difficulty and general image properties do not. With this approach, we try to open up new perspectives at the intersection of biological and computer vision research.