# HAND Me the Data: Fast Robot Adaptation via Hand Path Retrieval

arXiv: https://arxiv.org/abs/2505.20455 · HTML: https://mlsanity.com/paper/2505.20455/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 May 2025.

## Built on by (most cited first, 10 of 10)

- [Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisons](https://mlsanity.com/paper/2603.02115/) (2026)
- [Robot Learning from a Physical World Model](https://mlsanity.com/paper/2511.07416/) (2025)
- [Retrieve, Don't Retrain: Extending Vision Language Action Models to New Tasks at Test Time](https://mlsanity.com/paper/2606.15631/) (2026)
- [ReGIL: Retrieval-Guided Imitation Learning from a Single Demonstration](https://mlsanity.com/paper/2606.09381/) (2026)
- [LiMoDE: Rethinking Lifelong Robot Manipulation from a Mixture-of-Dynamic-Experts Perspective](https://mlsanity.com/paper/2606.26183/) (2026)
- [Retrieve in Time, Correct in Frequency](https://mlsanity.com/paper/2608.04527/) (2026)
- [RoboTok: A Scalable Data Engine for Internet Demonstration Video Retrieval and Dexterous Manipulation Learning](https://mlsanity.com/paper/2609.03199/) (2026)
- [ReWeight: Leveraging Human Data for VLA Post-Training via Demonstration Retrieval and Sample Weighting](https://mlsanity.com/paper/2609.13851/) (2026)
- [Learning Transferable World-Action Models from Task-Paired Human Videos](https://mlsanity.com/paper/2610.14351/) (2026)
- [Beyond the End-Effector: Learning Whole-Arm Contact Manipulation from Human Demonstrations](https://mlsanity.com/paper/2610.17802/) (2026)

## Abstract (arXiv)

We hand the community HAND, a simple and time-efficient method for teaching robots new manipulation tasks through human hand demonstrations. Instead of relying on task-specific robot demonstrations collected via teleoperation, HAND uses easy-to-provide hand demonstrations to retrieve relevant behaviors from task-agnostic robot play data. Using a visual tracking pipeline, HAND extracts the motion of the human hand from the hand demonstration and retrieves robot sub-trajectories in two stages: first filtering by visual similarity, then retrieving trajectories with similar behaviors to the hand. Fine-tuning a policy on the retrieved data enables real-time learning of tasks in under four minutes, without requiring calibrated cameras or detailed hand pose estimation. Experiments also show that HAND outperforms retrieval baselines by over 2x in average task success rates on real robots. Videos can be found at our project website: https://liralab.usc.edu/handretrieval/.