# Detecting Attended Visual Targets in Video

arXiv: https://arxiv.org/abs/2003.02501 · HTML: https://mlsanity.com/paper/2003.02501/

## Key facts

- Cited by 77 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Mar 2020.

## Built on by (most cited first, 30 of 77)

- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [In the Eye of Transformer: Global-Local Correlation for Egocentric Gaze Estimation](https://mlsanity.com/paper/2208.04464/) (2022)
- [End-to-End Human-Gaze-Target Detection with Transformers](https://mlsanity.com/paper/2203.10433/) (2022)
- [Gaze-LLE: Gaze Target Estimation via Large-Scale Learned Encoders](https://mlsanity.com/paper/2412.09586/) (2024)
- [ChildPlay: A New Benchmark for Understanding Children's Gaze Behaviour](https://mlsanity.com/paper/2307.01630/) (2023)
- [Automatic Gaze Analysis: A Survey of Deep Learning based Approaches](https://mlsanity.com/paper/2108.05479/) (2021)
- [A Modular Multimodal Architecture for Gaze Target Prediction: Application to Privacy-Sensitive Settings](https://mlsanity.com/paper/2307.05158/) (2023)
- [Weakly-Supervised Physically Unconstrained Gaze Estimation](https://mlsanity.com/paper/2105.09803/) (2021)
- [Object-aware Gaze Target Detection](https://mlsanity.com/paper/2307.09662/) (2023)
- [GOO: A Dataset for Gaze Object Prediction in Retail Environments](https://mlsanity.com/paper/2105.10793/) (2021)
- [Multimodal Across Domains Gaze Target Detection](https://mlsanity.com/paper/2208.10822/) (2022)
- [Patch-level Gaze Distribution Prediction for Gaze Following](https://mlsanity.com/paper/2211.11062/) (2022)
- [GaTector: A Unified Framework for Gaze Object Prediction](https://mlsanity.com/paper/2112.03549/) (2021)
- [Modeling Multimodal Social Interactions: New Challenges and Baselines with Densely Aligned Representations](https://mlsanity.com/paper/2403.02090/) (2024)
- [Understanding Embodied Reference with Touch-Line Transformer](https://mlsanity.com/paper/2210.05668/) (2022)
- [M2LADS: A System for Generating MultiModal Learning Analytics Dashboards in Open Education](https://mlsanity.com/paper/2305.12561/) (2023)
- [ViTGaze: Gaze Following with Interaction Features in Vision Transformers](https://mlsanity.com/paper/2403.12778/) (2024)
- [Exploring the Zero-Shot Capabilities of Vision-Language Models for Improving Gaze Following](https://mlsanity.com/paper/2406.03907/) (2024)
- [Human-Object Interaction Prediction in Videos through Gaze Following](https://mlsanity.com/paper/2306.03597/) (2023)
- [Towards Online Multi-Modal Social Interaction Understanding](https://mlsanity.com/paper/2503.19851/) (2025)
- [Biometrics and Behavior Analysis for Detecting Distractions in e-Learning](https://mlsanity.com/paper/2405.15434/) (2024)
- [Enhancing 3D Gaze Estimation in the Wild using Weak Supervision with Gaze Following Labels](https://mlsanity.com/paper/2502.20249/) (2025)
- [GazeVLM: A Vision-Language Model for Multi-Task Gaze Understanding](https://mlsanity.com/paper/2511.06348/) (2025)
- [UniGaze: Towards Universal Gaze Estimation via Large-scale Pre-Training](https://mlsanity.com/paper/2502.02307/) (2025)
- [A Gaze-grounded Visual Question Answering Dataset for Clarifying Ambiguous Japanese Questions](https://mlsanity.com/paper/2403.17545/) (2024)
- [Interaction-aware Joint Attention Estimation Using People Attributes](https://mlsanity.com/paper/2308.05382/) (2023)
- [Joint Gaze-Location and Gaze-Object Detection](https://mlsanity.com/paper/2308.13857/) (2023)
- [Multi-Modal Gaze Following in Conversational Scenarios](https://mlsanity.com/paper/2311.05669/) (2023)
- [TransGOP: Transformer-Based Gaze Object Prediction](https://mlsanity.com/paper/2402.13578/) (2024)
- [GazeHTA: End-to-end Gaze Target Detection with Head-Target Association](https://mlsanity.com/paper/2404.10718/) (2024)

## Abstract (arXiv)

We address the problem of detecting attention targets in video. Our goal is to identify where each person in each frame of a video is looking, and correctly handle the case where the gaze target is out-of-frame. Our novel architecture models the dynamic interaction between the scene and head features and infers time-varying attention targets. We introduce a new annotated dataset, VideoAttentionTarget, containing complex and dynamic patterns of real-world gaze behavior. Our experiments show that our model can effectively infer dynamic attention in videos. In addition, we apply our predicted attention maps to two social gaze behavior recognition tasks, and show that the resulting classifiers significantly outperform existing methods. We achieve state-of-the-art performance on three datasets: GazeFollow (static images), VideoAttentionTarget (videos), and VideoCoAtt (videos), and obtain the first results for automatically classifying clinically-relevant gaze behavior without wearable cameras or eye trackers.