# Hand-centric Human-to-Robot Trajectory Transfer from Video Demonstrations via Object Category-agnostic Temporal Localization

arXiv: https://arxiv.org/abs/2606.10743 · HTML: https://mlsanity.com/paper/2606.10743/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data.
- First posted on arXiv 9 Jun 2026.

## Placed in

- Offline RL and imitation learning from data / [Robot manipulation imitation learning](https://mlsanity.com/offlinerl/robot-manipulation-imitation-learning/)

## Built on by (most cited first, 1 of 1)

- [Human Demonstrations as Post-Training Data for Any Robot Body: Retargeting the Object, Not the Hand](https://mlsanity.com/paper/2610.20351/) (2026)

## Abstract (arXiv)

Human videos provide a scalable source of demonstrations for robot imitation learning. However, converting them into executable and semantically aligned robot trajectories requires determining when task-relevant interactions occur, how demonstrated human grasps should be retargeted across embodiments, and which post-contact motions should be reproduced. In this paper, we present textitHOCALT (Hand--Object Category--Agnostic Localization and Transfer), a hand-centric retargeting framework that unifies 3D hand motion reconstruction, temporal contact localization, and cross-embodiment trajectory generation. Given synchronized stereo videos, textitHOCALT identifies task-relevant contact intervals by jointly reasoning over reconstructed 3D hand articulation and category-agnostic object motion cues. These intervals serve as temporal anchors for initializing cross-embodiment transfer, where the demonstrated human grasps are retargeted into multi-modal robot grasp hypotheses. Within each interval, these hypotheses are then propagated along the demonstrated hand motion, yielding executable trajectories. Finally, we augment the transferred trajectories to generate diverse variants from a single demonstration. Across various tasks, HOCALT outperforms VLM-based temporal localization baselines, achieving higher replay success than existing retargeting approaches.