# Understanding Human Gaze Communication by Spatio-Temporal Graph Reasoning

arXiv: https://arxiv.org/abs/1909.02144 · HTML: https://mlsanity.com/paper/1909.02144/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Sep 2019.

## Built on by (most cited first, 30 of 40)

- [Multiscale Spatio-Temporal Graph Neural Networks for 3D Skeleton-Based Motion Prediction](https://mlsanity.com/paper/2108.11244/) (2021)
- [Gaze-LLE: Gaze Target Estimation via Large-Scale Learned Encoders](https://mlsanity.com/paper/2412.09586/) (2024)
- [ChildPlay: A New Benchmark for Understanding Children's Gaze Behaviour](https://mlsanity.com/paper/2307.01630/) (2023)
- [Automatic Gaze Analysis: A Survey of Deep Learning based Approaches](https://mlsanity.com/paper/2108.05479/) (2021)
- [Differentiable Multi-Granularity Human Representation Learning for Instance-Aware Human Semantic Parsing](https://mlsanity.com/paper/2103.04570/) (2021)
- [Skeleton-Parted Graph Scattering Networks for 3D Human Motion Prediction](https://mlsanity.com/paper/2208.00368/) (2022)
- [Position: Measure Dataset Diversity, Don't Just Claim It](https://mlsanity.com/paper/2407.08188/) (2024)
- [Neural-Logic Human-Object Interaction Detection](https://mlsanity.com/paper/2311.09817/) (2023)
- [IntentVizor: Towards Generic Query Guided Interactive Video Summarization](https://mlsanity.com/paper/2109.14834/) (2021)
- [Learning Triadic Belief Dynamics in Nonverbal Communication from Videos](https://mlsanity.com/paper/2104.02841/) (2021)
- [Graph Neural Network and Spatiotemporal Transformer Attention for 3D Video Object Detection from Point Clouds](https://mlsanity.com/paper/2207.12659/) (2022)
- [Scene Graph Generation with Role-Playing Large Language Models](https://mlsanity.com/paper/2410.15364/) (2024)
- [Human-Object Interaction Detection Collaborated with Large Relation-driven Diffusion Models](https://mlsanity.com/paper/2410.20155/) (2024)
- [End-to-end Video Gaze Estimation via Capturing Head-face-eye Spatial-temporal Interaction Context](https://mlsanity.com/paper/2310.18131/) (2023)
- [Nonverbal Interaction Detection](https://mlsanity.com/paper/2407.08133/) (2024)
- [Towards Online Multi-Modal Social Interaction Understanding](https://mlsanity.com/paper/2503.19851/) (2025)
- [DIFFVSGG: Diffusion-Driven Online Video Scene Graph Generation](https://mlsanity.com/paper/2503.13957/) (2025)
- [Self-supervised Social Relation Representation for Human Group Detection](https://mlsanity.com/paper/2203.03843/) (2022)
- [Bi-directional Object-context Prioritization Learning for Saliency Ranking](https://mlsanity.com/paper/2203.09416/) (2022)
- [How you feelin'? Learning Emotions and Mental States in Movie Scenes](https://mlsanity.com/paper/2304.05634/) (2023)
- [Multi-Modal Gaze Following in Conversational Scenarios](https://mlsanity.com/paper/2311.05669/) (2023)
- [Learning Human-Object Interaction as Groups](https://mlsanity.com/paper/2510.18357/) (2025)
- [Ano-Graph: Learning Normal Scene Contextual Graphs to Detect Video Anomalies](https://mlsanity.com/paper/2103.10502/) (2021)
- [UAHOI: Uncertainty-aware Robust Interaction Learning for HOI Detection](https://mlsanity.com/paper/2408.07430/) (2024)
- [OmniGaze: Reward-inspired Generalizable Gaze Estimation In The Wild](https://mlsanity.com/paper/2510.13660/) (2025)
- [ERA: Entity Relationship Aware Video Summarization with Wasserstein GAN](https://mlsanity.com/paper/2109.02625/) (2021)
- [MGTR: End-to-End Mutual Gaze Detection with Transformer](https://mlsanity.com/paper/2209.10930/) (2022)
- [A Novel Framework for Multi-Person Temporal Gaze Following and Social Gaze Prediction](https://mlsanity.com/paper/2403.10511/) (2024)
- [Preemptive Motion Planning for Human-to-Robot Indirect Placement Handovers](https://mlsanity.com/paper/2203.00156/) (2022)
- [GRASP: Learning to Ground Social Reasoning in Multi-Person Non-Verbal Interactions](https://mlsanity.com/paper/2605.15764/) (2026)

## Abstract (arXiv)

This paper addresses a new problem of understanding human gaze communication in social videos from both atomic-level and event-level, which is significant for studying human social interactions. To tackle this novel and challenging problem, we contribute a large-scale video dataset, VACATION, which covers diverse daily social scenes and gaze communication behaviors with complete annotations of objects and human faces, human attention, and communication structures and labels in both atomic-level and event-level. Together with VACATION, we propose a spatio-temporal graph neural network to explicitly represent the diverse gaze interactions in the social scenes and to infer atomic-level gaze communication by message passing. We further propose an event network with encoder-decoder structure to predict the event-level gaze communication. Our experiments demonstrate that the proposed model improves various baselines significantly in predicting the atomic-level and event-level gaze