# H2R: A Human-to-Robot Data Augmentation for Robot Pre-training from Videos

arXiv: https://arxiv.org/abs/2505.11920 · HTML: https://mlsanity.com/paper/2505.11920/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 May 2025.

## Built on by (most cited first, 23 of 23)

- [EgoWAM: World Action Models Beyond Pixels with In-the-Wild Egocentric Human Data](https://mlsanity.com/paper/2607.08436/) (2026)
- [HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videos](https://mlsanity.com/paper/2605.24934/) (2026)
- [X-Humanoid: Robotize Human Videos to Generate Humanoid Videos at Scale](https://mlsanity.com/paper/2512.04537/) (2025)
- [OXE-AugE: A Large-Scale Robot Augmentation of OXE for Scaling Cross-Embodiment Policy Learning](https://mlsanity.com/paper/2512.13100/) (2025)
- [GuidedVLA: Specifying Task-Relevant Factors via Plug-and-Play Action Attention Specialization](https://mlsanity.com/paper/2605.12369/) (2026)
- [ACE-Ego-0: Unifying Egocentric Human and Robotic Data for VLA Pretraining](https://mlsanity.com/paper/2606.17200/) (2026)
- [H2R-Grounder: A Paired-Data-Free Paradigm for Translating Human Interaction Videos into Physically Grounded Robot Videos](https://mlsanity.com/paper/2512.09406/) (2025)
- [Vision-Language-Action in Robotics: A Survey of Datasets, Benchmarks, and Data Engines](https://mlsanity.com/paper/2604.23001/) (2026)
- [Data Pyramid for Embodied Manipulation: A Survey](https://mlsanity.com/paper/2607.24744/) (2026)
- [Robot Learning from Human Videos: A Survey](https://mlsanity.com/paper/2604.27621/) (2026)
- [Visual Generation in the New Era: An Evolution from Atomic Mapping to Agentic World Modeling](https://mlsanity.com/paper/2604.28185/) (2026)
- [MolmoMotion: Forecasting Point Trajectories in 3D with Language Instruction](https://mlsanity.com/paper/2606.18558/) (2026)
- [Cloak: Zero-Shot Cross-Embodiment Manipulation by Masking the End-Effector from the VLA](https://mlsanity.com/paper/2606.22836/) (2026)
- [HandEdit: A Unified Benchmark for Egocentric Human-to-Robot Dexterous Hand Image Editing](https://mlsanity.com/paper/2608.12122/) (2026)
- [H2R-Bench: Benchmarking Human-to-Robot Manipulation Video Generation in World Models](https://mlsanity.com/paper/2608.13049/) (2026)
- [RynnWorld-Teleop: An Action-Conditioned World Model for Digital Teleoperation](https://mlsanity.com/paper/2607.06558/) (2026)
- [TrajBooster: Boosting Humanoid Whole-Body Manipulation via Trajectory-Centric Learning](https://mlsanity.com/paper/2509.11839/) (2025)
- [EasyMimic: A Low-Cost Framework for Robot Imitation Learning from Human Videos](https://mlsanity.com/paper/2602.11464/) (2026)
- [LIDEA: Human-to-Robot Imitation Learning via Implicit Feature Distillation and Explicit Geometry Alignment](https://mlsanity.com/paper/2604.10677/) (2026)
- [HiPHI: A Large-Scale Benchmark for High-Precision Human Motion and Object-Interaction](https://mlsanity.com/paper/2608.16222/) (2026)
- [HuRo: Robotizing Human Videos for Scalable VLA Pretraining](https://mlsanity.com/paper/2609.10706/) (2026)
- [Skel-WAM: A Hand-Skeleton-Conditioned World Action Model for Human-to-Robot Manipulation Transfer](https://mlsanity.com/paper/2609.21514/) (2026)
- [DexStitch: Learning and Composing Dexterous Skills from Human Videos via Contact-Aware Robotic Inpainting](https://mlsanity.com/paper/2610.17789/) (2026)

## Abstract (arXiv)

Large-scale pre-training using egocentric human videos has proven effective for robot learning. However, the models pre-trained on such data can be suboptimal for robot learning due to the significant visual gap between human hands and those of different robots. To remedy this, we propose H2R, a human-to-robot data augmentation pipeline that converts egocentric human videos into robot-centric visual data. H2R estimates human hand pose from videos, retargets the motion to simulated robotic arms, removes human limbs via segmentation and inpainting, and composites rendered robot embodiments into the original frames with camera-aligned geometry. This process explicitly bridges the visual gap between human and robot embodiments during pre-training. We apply H2R to augment large-scale egocentric human video datasets such as Ego4D and SSv2. To verify the effectiveness of the augmentation pipeline, we introduce a CLIP-based image-text similarity metric that quantitatively evaluates the semantic fidelity of robot-rendered frames to the original human actions. We evaluate H2R through comprehensive experiments in both simulation and real-world settings. In simulation, H2R consistently improves downstream success rates across four benchmark suites-Robomimic, RLBench, PushT, and CortexBench-yielding gains of 1.3%-10.2% across different visual encoders and policy learning methods. In real-world experiments, H2R improves performance on UR5 and dual-arm Franka/UR5 manipulation platforms, achieving 3.3%-23.3% success rate gains across gripper-based, dexterous, and bimanual tasks. We further demonstrate the potential of H2R in cross-embodiment generalization and its compatibility with vision-language-action models. These results indicate that H2R improves the generalization ability of robotic policies by mitigating the visual discrepancies between human and robot domains.