Sharingan: A Transformer-based Architecture for Gaze Following
Not on the map. Why?
Abstract (arXiv)
Gaze is a powerful form of non-verbal communication and social interaction that humans develop from an early age. As such, modeling this behavior is an important task that can benefit a broad set of application domains ranging from robotics to sociology. In particular, Gaze Following is defined as the prediction of the pixel-wise 2D location where a person in the image is looking. Prior efforts in this direction have focused primarily on CNN-based architectures to perform the task. In this paper, we introduce a novel transformer-based architecture for 2D gaze prediction. We experiment with 2 variants: the first one retains the same task formulation of predicting a gaze heatmap for one person at a time, while the second one casts the problem as a 2D point regression and allows us to perform multi-person gaze prediction with a single forward pass. This new architecture achieves state-of-the-art results on the GazeFollow and VideoAttentionTarget datasets. The code for this paper will be made publicly available.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 31 of its 52 references that are in the map. First posted on arXiv 1 Oct 2023.
Builds on
31 of 52 references.
References · 31
- 2017Attention Is All You Needcited by 60,361
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2014Microsoft COCO: Common Objects in Contextcited by 17,608
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
- 2020End-to-End Object Detection with Transformerscited by 7,414
- 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
- 2019A Comprehensive Survey on Graph Neural Networkscited by 2,919
- 2021Vision Transformers for Dense Predictioncited by 1,803
- 2021ViViT: A Video Vision Transformercited by 1,620
- 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
19 more
- 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 291
- 2017Learning to Detect Human-Object Interactionscited by 240
- 2017Detecting and Recognizing Human-Object Interactionscited by 199
- 2021HOTR: End-to-End Human-Object Interaction Detection with Transformerscited by 183
- 2018Learning Human-Object Interactions by Graph Parsing Neural Networkscited by 176
- 2019Gaze360: Physically Unconstrained Gaze Estimation in the Wildcited by 174
- 2021End-to-End Human Object Interaction Detection with HOI Transformercited by 135
- 2018iCAN: Instance-Centric Attention Network for Human-Object Interaction Detectioncited by 127
- 2020Detecting Attended Visual Targets in Videocited by 77
- 2019Effectiveness of self-supervised pre-training for speech recognitioncited by 72
- 2018Connecting Gaze, Scene, and Attention: Generalized Attention Estimation via Joint Modeling of Gaze and Scene Saliencycited by 53
- 2022End-to-End Human-Gaze-Target Detection with Transformerscited by 47
- 2022Human-Object Interaction Detection via Disentangled Transformercited by 40
- 2019Believe It or Not, We Know What You Are Looking at!cited by 38
- 2019LAEO-Net: revisiting people Looking At Each Other in videoscited by 34
- 2023A Modular Multimodal Architecture for Gaze Target Prediction: Application to Privacy-Sensitive Settingscited by 31
- 2021GOO: A Dataset for Gaze Object Prediction in Retail Environmentscited by 26
- 2022Iwin: Human-Object Interaction Detection via Transformer with Irregular Windowscited by 19
- 2022Detecting Human-to-Human-or-Object (H2O) Interactions with DIABOLOcited by 2
Built on by
Citing papers per year: 2024: 1.
1 paper.