# GazeMotion: Gaze-guided Human Motion Forecasting

arXiv: https://arxiv.org/abs/2403.09885 · HTML: https://mlsanity.com/paper/2403.09885/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 23 references that are in the map.
- First posted on arXiv 14 Mar 2024.

## Builds on (most cited first, 13 of 13)

- [On human motion prediction using recurrent neural networks](https://mlsanity.com/paper/1705.02445/) (2017)
- [Learning Trajectory Dependencies for Human Motion Prediction](https://mlsanity.com/paper/1908.05436/) (2019)
- [Aria Digital Twin: A New Benchmark Dataset for Egocentric 3D Machine Perception](https://mlsanity.com/paper/2306.06362/) (2023)
- [Back to MLP: A Simple Baseline for Human Motion Prediction](https://mlsanity.com/paper/2207.01567/) (2022)
- [GIMO: Gaze-Informed Human Motion Prediction in Context](https://mlsanity.com/paper/2204.09443/) (2022)
- [Progressively Generating Better Initial Guesses Towards Next Stages for High-Quality Human Motion Prediction](https://mlsanity.com/paper/2203.16051/) (2022)
- [Multi-level Motion Attention for Human Motion Prediction](https://mlsanity.com/paper/2106.09300/) (2021)
- [MoGaze: A Dataset of Full-Body Motions that Includes Workspace Geometry and Eye-Gaze](https://mlsanity.com/paper/2011.11552/) (2020)
- [Action Anticipation: Reading the Intentions of Humans and Robots](https://mlsanity.com/paper/1802.02788/) (2018)
- [Intention estimation from gaze and motion features for human-robot shared-control object manipulation](https://mlsanity.com/paper/2208.08688/) (2022)
- [Memory-based gaze prediction in deep imitation learning for robot manipulation](https://mlsanity.com/paper/2202.04877/) (2022)
- [Pose2Gaze: Eye-body Coordination during Daily Activities for Gaze Prediction from Full-body Poses](https://mlsanity.com/paper/2312.12042/) (2023)
- [Hierarchical Human-Motion Prediction and Logic-Geometric Programming for Minimal Interference Human-Robot Tasks](https://mlsanity.com/paper/2104.08137/) (2021)

## Built on by (most cited first, 5 of 5)

- [Gaze-guided Hand-Object Interaction Synthesis: Dataset and Method](https://mlsanity.com/paper/2403.16169/) (2024)
- [HaHeAE: Learning Generalisable Joint Representations of Human Hand and Head Movements in Extended Reality](https://mlsanity.com/paper/2410.16430/) (2024)
- [Learning Predictive Visuomotor Coordination](https://mlsanity.com/paper/2503.23300/) (2025)
- [Seeing My Future: Predicting Situated Interaction Behavior in Virtual Reality](https://mlsanity.com/paper/2510.10742/) (2025)
- [HAGI++: Head-Assisted Gaze Imputation and Generation](https://mlsanity.com/paper/2511.02468/) (2025)

## Abstract (arXiv)

We present GazeMotion, a novel method for human motion forecasting that combines information on past human poses with human eye gaze. Inspired by evidence from behavioural sciences showing that human eye and body movements are closely coordinated, GazeMotion first predicts future eye gaze from past gaze, then fuses predicted future gaze and past poses into a gaze-pose graph, and finally uses a residual graph convolutional network to forecast body motion. We extensively evaluate our method on the MoGaze, ADT, and GIMO benchmark datasets and show that it outperforms state-of-the-art methods by up to 7.4% improvement in mean per joint position error. Using head direction as a proxy to gaze, our method still achieves an average improvement of 5.5%. We finally report an online user study showing that our method also outperforms prior methods in terms of perceived realism. These results show the significant information content available in eye gaze for human motion forecasting as well as the effectiveness of our method in exploiting this information.