Human Gaze Boosts Object-Centered Representation Learning
Not on the map. Why?
Abstract (arXiv)
Recent self-supervised learning (SSL) models trained on human-like egocentric visual inputs substantially underperform on image recognition tasks compared to humans. These models train on raw, uniform visual inputs collected from head-mounted cameras. This is different from humans, as the anatomical structure of the retina and visual cortex relatively amplifies the central visual information, i.e. around humans' gaze location. This selective amplification in humans likely aids in forming object-centered visual representations. Here, we investigate whether focusing on central visual information boosts egocentric visual object learning. We simulate 5-months of egocentric visual experience using the large-scale Ego4D dataset and generate gaze locations with a human gaze prediction model. To account for the importance of central vision in humans, we crop the visual area around the gaze location. Finally, we train a time-based SSL model on these modified inputs. Our experiments demonstrate that focusing on central vision leads to better object-centered representations. Our analysis shows that the SSL model leverages the temporal dynamics of the gaze movements to build stronger visual representations. Overall, our work marks a significant step toward bio-inspired learning of visual representations.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 44 of its 67 references that are in the map. First posted on arXiv 6 Jan 2025.
Builds on
44 of 67 references.
References · 44
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
- 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
- 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
- 2013Describing Textures in the Wildcited by 2,125
- 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
- 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
- 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
- 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
- 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
32 more
- 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 694
- 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 459
- 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 395
- 2020Noise or Signal: The Role of Image Backgrounds in Object Recognitioncited by 343
- 2017CORe50: a New Dataset and Benchmark for Continuous Object Recognitioncited by 289
- 2020Space-Time Correspondence as a Contrastive Random Walkcited by 239
- 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 218
- 2021Partial success in closing the gap between human and machine visioncited by 185
- 2021Solo-learn: A Library of Self-supervised Methods for Visual Representation Learningcited by 116
- 2024Nymeria: A Massive Collection of Multimodal Egocentric Daily Motion in the Wildcited by 114
- 2021Rethinking Self-supervised Correspondence Learning: A Video Frame-level Similarity Perspectivecited by 81
- 2020Self-supervised learning through the eyes of a childcited by 61
- 2020Watching the World Go By: Representation Learning from Unlabeled Videoscited by 59
- 2022Mugs: A Multi-Granular Self-Supervised Learning Frameworkcited by 54
- 2022In the Eye of Transformer: Global-Local Correlation for Egocentric Gaze Estimationcited by 51
- 2015Learning image representations tied to ego-motioncited by 42
- 2023Is ImageNet worth 1 video? Learning strong image encoders from 1 long unlabelled videocited by 40
- 2023Time Does Tell: Self-Supervised Time-Tuning of Dense Image Representationscited by 35
- 2015Slow and steady feature analysis: higher order temporal coherence in videocited by 28
- 2023Learning high-level visual representations from a child's perspective without strong inductive biasescited by 22
- 2022Self-supervised video pretraining yields robust and more human-aligned visual representationscited by 20
- 2024The BabyView dataset: High-resolution egocentric videos of infants' and young children's everyday experiencescited by 17
- 2022Time to augment self-supervised visual representation learningcited by 15
- 2021On the use of Cortical Magnification and Saccades as Biological Proxies for Data Augmentationcited by 12
- 2024Self-supervised visual learning from interactions with objectscited by 10
- 2023Are Vision Transformers More Data Hungry Than Newborn Visual Systems?cited by 7
- 2024Self-supervised learning of video representations from a child's perspectivecited by 6
- 2024Learning Object Semantic Similarity with Self-Supervisioncited by 5
- 2023A Computational Account Of Self-Supervised Visual Learning From Egocentric Object Playcited by 4
- 2023Scaling may be all you need for achieving human-level object recognition capacity with human-like visual experiencecited by 4
- 2024The Visual Experience Dataset: Over 200 Recorded Hours of Integrated Eye Movement, Odometry, and Egocentric Videocited by 4
- 2024HVM-1: Large-scale video models pretrained with nearly 5000 hours of human-like video datacited by 2
Built on by
Citing papers per year: 2025: 1.
1 paper.