# HaHeAE: Learning Generalisable Joint Representations of Human Hand and Head Movements in Extended Reality

arXiv: https://arxiv.org/abs/2410.16430 · HTML: https://mlsanity.com/paper/2410.16430/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 72 references that are in the map.
- First posted on arXiv 21 Oct 2024.

## Builds on (most cited first, 30 of 33)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [NVAE: A Deep Hierarchical Variational Autoencoder](https://mlsanity.com/paper/2007.03898/) (2020)
- [Diffusion Autoencoders: Toward a Meaningful and Decodable Representation](https://mlsanity.com/paper/2111.15640/) (2021)
- [Autoregressive Denoising Diffusion Models for Multivariate Probabilistic Time Series Forecasting](https://mlsanity.com/paper/2101.12072/) (2021)
- [Aria Digital Twin: A New Benchmark Dataset for Egocentric 3D Machine Perception](https://mlsanity.com/paper/2306.06362/) (2023)
- [GIMO: Gaze-Informed Human Motion Prediction in Context](https://mlsanity.com/paper/2204.09443/) (2022)
- [Progressively Generating Better Initial Guesses Towards Next Stages for High-Quality Human Motion Prediction](https://mlsanity.com/paper/2203.16051/) (2022)
- [Causal Recurrent Variational Autoencoder for Medical Time Series Generation](https://mlsanity.com/paper/2301.06574/) (2023)
- [Screen2Vec: Semantic Embedding of GUI Screens and GUI Components](https://mlsanity.com/paper/2101.11103/) (2021)
- [A Recurrent Variational Autoencoder for Speech Enhancement](https://mlsanity.com/paper/1910.10942/) (2019)
- [Unique Identification of 50,000+ Virtual Reality Users from Head & Hand Motion Data](https://mlsanity.com/paper/2302.08927/) (2023)
- [Action Anticipation: Reading the Intentions of Humans and Robots](https://mlsanity.com/paper/1802.02788/) (2018)
- [HOIMotion: Forecasting Human Motion During Human-Object Interactions Using Egocentric 3D Object Bounding Boxes](https://mlsanity.com/paper/2407.02633/) (2024)
- [Structure-aware Visualization Retrieval](https://mlsanity.com/paper/2202.05960/) (2022)
- [Intention estimation from gaze and motion features for human-robot shared-control object manipulation](https://mlsanity.com/paper/2208.08688/) (2022)
- [GazeMoDiff: Gaze-guided Diffusion Model for Stochastic Human Motion Prediction](https://mlsanity.com/paper/2312.12090/) (2023)
- [Federated Learning for Appearance-based Gaze Estimation in the Wild](https://mlsanity.com/paper/2211.07330/) (2022)
- [Pose2Gaze: Eye-body Coordination during Daily Activities for Gaze Prediction from Full-body Poses](https://mlsanity.com/paper/2312.12042/) (2023)
- [WESPER: Zero-shot and Realtime Whisper to Normal Voice Conversion for Whisper-based Speech Interactions](https://mlsanity.com/paper/2303.01639/) (2023)
- [LipLearner: Customizable Silent Speech Interactions on Mobile Devices](https://mlsanity.com/paper/2302.05907/) (2023)
- [GazeMotion: Gaze-guided Human Motion Forecasting](https://mlsanity.com/paper/2403.09885/) (2024)
- [The Elements of Visual Art Recommendation: Learning Latent Semantic Representations of Paintings](https://mlsanity.com/paper/2303.08182/) (2023)

## Built on by (most cited first, 2 of 2)

- [HOIGaze: Gaze Estimation During Hand-Object Interactions in Extended Reality Exploiting Eye-Hand-Head Coordination](https://mlsanity.com/paper/2504.19828/) (2025)
- [HAGI++: Head-Assisted Gaze Imputation and Generation](https://mlsanity.com/paper/2511.02468/) (2025)

## Abstract (arXiv)

Human hand and head movements are the most pervasive input modalities in extended reality (XR) and are significant for a wide range of applications. However, prior works on hand and head modelling in XR only explored a single modality or focused on specific applications. We present HaHeAE - a novel self-supervised method for learning generalisable joint representations of hand and head movements in XR. At the core of our method is an autoencoder (AE) that uses a graph convolutional network-based semantic encoder and a diffusion-based stochastic encoder to learn the joint semantic and stochastic representations of hand-head movements. It also features a diffusion-based decoder to reconstruct the original signals. Through extensive evaluations on three public XR datasets, we show that our method 1) significantly outperforms commonly used self-supervised methods by up to 74.0% in terms of reconstruction quality and is generalisable across users, activities, and XR environments, 2) enables new applications, including interpretable hand-head cluster identification and variable hand-head movement generation, and 3) can serve as an effective feature extractor for downstream tasks. Together, these results demonstrate the effectiveness of our method and underline the potential of self-supervised methods for jointly modelling hand-head behaviours in extended reality.