# Data-driven Head Motion Generation through Natural Gaze-Head Coordination

arXiv: https://arxiv.org/abs/2605.25810 · HTML: https://mlsanity.com/paper/2605.25810/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 18 references that are in the map.
- First posted on arXiv 25 May 2026.

## Builds on (most cited first, 10 of 10)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)](https://mlsanity.com/paper/1703.07332/) (2017)
- [LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control](https://mlsanity.com/paper/2407.03168/) (2024)
- [DLow: Diversifying Latent Flows for Diverse Human Motion Prediction](https://mlsanity.com/paper/2003.08386/) (2020)
- [CelebV-Text: A Large-Scale Facial Text-Video Dataset](https://mlsanity.com/paper/2303.14717/) (2023)
- [Self-Learning Transformations for Improving Gaze and Head Redirection](https://mlsanity.com/paper/2010.12307/) (2020)
- [Gaze-in-wild: A dataset for studying eye and head coordination in everyday activities](https://mlsanity.com/paper/1905.13146/) (2019)
- [Learning-by-Novel-View-Synthesis for Full-Face Appearance-Based 3D Gaze Estimation](https://mlsanity.com/paper/2201.07927/) (2022)
- [GazeNeRF: 3D-Aware Gaze Redirection with Neural Radiance Fields](https://mlsanity.com/paper/2212.04823/) (2022)
- [ReDirTrans: Latent-to-Latent Translation for Gaze and Head Redirection](https://mlsanity.com/paper/2305.11452/) (2023)

## Abstract (arXiv)

We present the first data-driven approach to model temporal gaze-head coordination from large-scale in-the-wild facial videos. To obtain training data for generalizable learning, we propose an automatic pipeline that extracts natural yet diverse gaze and head motions with off-the-shelf appearance-based gaze estimators. To capture the probabilistic correlation and temporal dynamics of gaze-head coordination, we build our model on a generative conditional Variational Autoencoder for plausible yet diverse gaze-conditioned head motion generations. We further apply our framework to gaze-controlled facial video generation, where we enable video generation with natural and realistic head motion correlated to the input gaze - an aspect that has not been emphasized before. Human evaluation and quantitative comparisons demonstrate our method's effectiveness and validate our design choices, with evaluators showing statistically significant preference for our approach over baseline methods.