# Pose2Gaze: Eye-body Coordination during Daily Activities for Gaze Prediction from Full-body Poses

arXiv: https://arxiv.org/abs/2312.12042 · HTML: https://mlsanity.com/paper/2312.12042/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 59 references that are in the map.
- First posted on arXiv 19 Dec 2023.

## Builds on (most cited first, 13 of 13)

- [Aria Digital Twin: A New Benchmark Dataset for Egocentric 3D Machine Perception](https://mlsanity.com/paper/2306.06362/) (2023)
- [History Repeats Itself: Human Motion Prediction via Motion Attention](https://mlsanity.com/paper/2007.11755/) (2020)
- [Robots Learn Social Skills: End-to-End Learning of Co-Speech Gesture Generation for Humanoid Robots](https://mlsanity.com/paper/1810.12541/) (2018)
- [Text2Gestures: A Transformer-Based Network for Generating Emotive Body Gestures for Virtual Agents](https://mlsanity.com/paper/2101.11101/) (2021)
- [GIMO: Gaze-Informed Human Motion Prediction in Context](https://mlsanity.com/paper/2204.09443/) (2022)
- [Progressively Generating Better Initial Guesses Towards Next Stages for High-Quality Human Motion Prediction](https://mlsanity.com/paper/2203.16051/) (2022)
- [Analyzing Input and Output Representations for Speech-Driven Gesture Generation](https://mlsanity.com/paper/1903.03369/) (2019)
- [ChoreoNet: Towards Music to Dance Synthesis with Choreographic Action Unit](https://mlsanity.com/paper/2009.07637/) (2020)
- [Gaze-in-wild: A dataset for studying eye and head coordination in everyday activities](https://mlsanity.com/paper/1905.13146/) (2019)
- [MoGaze: A Dataset of Full-Body Motions that Includes Workspace Geometry and Eye-Gaze](https://mlsanity.com/paper/2011.11552/) (2020)
- [Action Anticipation: Reading the Intentions of Humans and Robots](https://mlsanity.com/paper/1802.02788/) (2018)
- [Forecasting User Attention During Everyday Mobile Interactions Using Device-Integrated and Wearable Sensors](https://mlsanity.com/paper/1801.06011/) (2018)
- [EVA: Generating Emotional Behavior of Virtual Agents using Expressive Features of Gait and Gaze](https://mlsanity.com/paper/1907.02102/) (2019)

## Built on by (most cited first, 7 of 7)

- [HOIMotion: Forecasting Human Motion During Human-Object Interactions Using Egocentric 3D Object Bounding Boxes](https://mlsanity.com/paper/2407.02633/) (2024)
- [GazeMotion: Gaze-guided Human Motion Forecasting](https://mlsanity.com/paper/2403.09885/) (2024)
- [HaHeAE: Learning Generalisable Joint Representations of Human Hand and Head Movements in Extended Reality](https://mlsanity.com/paper/2410.16430/) (2024)
- [Seeing My Future: Predicting Situated Interaction Behavior in Virtual Reality](https://mlsanity.com/paper/2510.10742/) (2025)
- [HAGI++: Head-Assisted Gaze Imputation and Generation](https://mlsanity.com/paper/2511.02468/) (2025)
- [GazeD: Context-Aware Diffusion for Accurate 3D Gaze Estimation](https://mlsanity.com/paper/2601.12948/) (2026)
- [SynAgent: Generalizable Cooperative Humanoid Manipulation via Solo-to-Cooperative Agent Synergy](https://mlsanity.com/paper/2604.18557/) (2026)

## Abstract (arXiv)

Human eye gaze plays a significant role in many virtual and augmented reality (VR/AR) applications, such as gaze-contingent rendering, gaze-based interaction, or eye-based activity recognition. However, prior works on gaze analysis and prediction have only explored eye-head coordination and were limited to human-object interactions. We first report a comprehensive analysis of eye-body coordination in various human-object and human-human interaction activities based on four public datasets collected in real-world (MoGaze), VR (ADT), as well as AR (GIMO and EgoBody) environments. We show that in human-object interactions, e.g. pick and place, eye gaze exhibits strong correlations with full-body motion while in human-human interactions, e.g. chat and teach, a person's gaze direction is correlated with the body orientation towards the interaction partner. Informed by these analyses we then present Pose2Gaze, a novel eye-body coordination model that uses a convolutional neural network and a spatio-temporal graph convolutional neural network to extract features from head direction and full-body poses, respectively, and then uses a convolutional neural network to predict eye gaze. We compare our method with state-of-the-art methods that predict eye gaze only from head movements and show that Pose2Gaze outperforms these baselines with an average improvement of 24.0% on MoGaze, 10.1% on ADT, 21.3% on GIMO, and 28.6% on EgoBody in mean angular error, respectively. We also show that our method significantly outperforms prior methods in the sample downstream task of eye-based activity recognition. These results underline the significant information content available in eye-body coordination during daily activities and open up a new direction for gaze prediction.