# HAP: A Hand-Driven Active Perception Framework for Egocentric Head Motion Prediction

arXiv: https://arxiv.org/abs/2609.18548 · HTML: https://mlsanity.com/paper/2609.18548/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 34 of its 37 references that are in the map.
- First posted on arXiv 16 Sep 2026.

## Builds on (most cited first, 30 of 34)

- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAM](https://mlsanity.com/paper/2007.11898/) (2020)
- [Depth Anything 3: Recovering the Visual Space from Any Views](https://mlsanity.com/paper/2511.10647/) (2025)
- [Principal Neighbourhood Aggregation for Graph Nets](https://mlsanity.com/paper/2004.05718/) (2020)
- [YOLOv11: An Overview of the Key Architectural Enhancements](https://mlsanity.com/paper/2410.17725/) (2024)
- [A Recurrent Latent Variable Model for Sequential Data](https://mlsanity.com/paper/1506.02216/) (2015)
- [AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting](https://mlsanity.com/paper/2103.14023/) (2021)
- [SAM 3D: 3Dfy Anything in Images](https://mlsanity.com/paper/2511.16624/) (2025)
- [Deep Kalman Filters](https://mlsanity.com/paper/1511.05121/) (2015)
- [WiLoR: End-to-end 3D Hand Localization and Reconstruction in-the-wild](https://mlsanity.com/paper/2409.12259/) (2024)
- [Disentangled Sequential Autoencoder](https://mlsanity.com/paper/1803.02991/) (2018)
- [Sequential Neural Models with Stochastic Layers](https://mlsanity.com/paper/1605.07571/) (2016)
- [Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videos](https://mlsanity.com/paper/2204.01696/) (2022)
- [An Outlook into the Future of Egocentric Vision](https://mlsanity.com/paper/2308.07123/) (2023)
- [AntGPT: Can Large Language Models Help Long-term Action Anticipation from Videos?](https://mlsanity.com/paper/2307.16368/) (2023)
- [Language-Grounded Dynamic Scene Graphs for Interactive Object Search with Mobile Manipulation](https://mlsanity.com/paper/2403.08605/) (2024)
- [UniTraj: A Unified Framework for Scalable Vehicle Trajectory Prediction](https://mlsanity.com/paper/2403.15098/) (2024)
- [Action Scene Graphs for Long-Form Understanding of Egocentric Videos](https://mlsanity.com/paper/2312.03391/) (2023)
- [A Recurrent Variational Autoencoder for Speech Enhancement](https://mlsanity.com/paper/1910.10942/) (2019)
- [Vision in Action: Learning Active Perception from Human Demonstrations](https://mlsanity.com/paper/2506.15666/) (2025)
- [Uncertainty-aware State Space Transformer for Egocentric 3D Hand Trajectory Forecasting](https://mlsanity.com/paper/2307.08243/) (2023)
- [Egocentric Prediction of Action Target in 3D](https://mlsanity.com/paper/2203.13116/) (2022)
- [SMEMO: Social Memory for Trajectory Forecasting](https://mlsanity.com/paper/2203.12446/) (2022)
- [Diff-IP2D: Diffusion-Based Hand-Object Interaction Prediction on Egocentric Videos](https://mlsanity.com/paper/2405.04370/) (2024)
- [MADiff: Motion-Aware Mamba Diffusion Models for Hand Trajectory Prediction on Egocentric Videos](https://mlsanity.com/paper/2409.02638/) (2024)
- [Unified Uncertainty-Aware Diffusion for Multi-Agent Trajectory Modeling](https://mlsanity.com/paper/2503.18589/) (2025)
- [Active-Perceptive Motion Generation for Mobile Manipulation](https://mlsanity.com/paper/2310.00433/) (2023)
- [EgoPAT3Dv2: Predicting 3D Action Target from 2D Egocentric Vision for Human-Robot Interaction](https://mlsanity.com/paper/2403.05046/) (2024)
- [Uni-Hand: Universal Hand Motion Forecasting in Egocentric Views](https://mlsanity.com/paper/2511.12878/) (2025)

## Abstract (arXiv)

Egocentric motion forecasting has primarily focused on hands and manipulated objects, leaving future human head motion comparatively underexplored. During manipulation, the head both redirects perception toward the target to acquire task-relevant evidence and coordinates with body and hand motion. We therefore formulate future six Degree of Freedom (6-DoF) head-motion prediction conditioned on observed hand motion and inferred target context, and propose HAP, a Hand-Driven Active Perception framework. HAP infers confidence for each target object from observed hand motion and object geometry. Then constructs a dynamic Predictive Target-Centric Amodal Occlusion Graph (P-TAOG) representing current and potential occlusion among candidate objects. Directed graph and causal temporal reasoning encode the evolving target conditioned perceptual state, which is fused with hand and head motion history. A horizon-wise gate then blends the learned trajectory with a constant velocity prior. We further introduce Bottle, an egocentric RGB-D dataset of object manipulation toward specified targets, with coordinated head and hand motion under changing target visibility. Experiments on the public dataset and Bottle show that HAP achieves lower head motion prediction errors than representative baselines, supporting the value of hand driven intention and dynamic occlusion reasoning for anticipating human head motion. Code will be released at https://HAP-ego.github.io/HAP.