Hand-centric Human-to-Robot Trajectory Transfer from Video Demonstrations via Object Category-agnostic Temporal Localization

Human videos provide a scalable source of demonstrations for robot imitation learning. However, converting them into executable and semantically aligned robot trajectories requires determining when task-relevant interactions occur, how demonstrated human grasps should be retargeted across embodiments, and which post-contact motions should be reproduced. In this paper, we present textitHOCALT (Hand--Object Category--Agnostic Localization and Transfer), a hand-centric retargeting framework that unifies 3D hand motion reconstruction, temporal contact localization, and cross-embodiment trajectory generation. Given synchronized stereo videos, textitHOCALT identifies task-relevant contact intervals by jointly reasoning over reconstructed 3D hand articulation and category-agnostic object motion cues. These intervals serve as temporal anchors for initializing cross-embodiment transfer, where the demonstrated human grasps are retargeted into multi-modal robot grasp hypotheses. Within each interval, these hypotheses are then propagated along the demonstrated hand motion, yielding executable trajectories. Finally, we augment the transferred trajectories to generate diverse variants from a single demonstration. Across various tasks, HOCALT outperforms VLM-based temporal localization baselines, achieving higher replay success than existing retargeting approaches.