What Matters When Cotraining Robot Manipulation Policies on Everyday Human Videos?
Not on the map. Why?
Abstract (arXiv)
Human video datasets used for cotraining robot manipulation policies largely consist of curated demonstrations where motions are orchestrated to resemble robot behavior and 3D hand poses are captured with specialized hardware. A more plentiful source of data is everyday Internet video, but it is an open question what factors enable transfer from such videos to robots. We investigate this using a new dataset of 532 human videos with 28 hours of high-quality triangulated hand labels and natural motions. We find that hand pose quality affects transfer, but even with accurate hands, the inherent motion gap hinders transfer unless the vision and policy networks specialize to each embodiment. Our cotraining recipe yields consistent improvements, with an absolute success rate gain of 29.7% in the low-robot-data regime across six manipulation tasks.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 32 of its 38 references that are in the map. First posted on arXiv 4 Jun 2026.
Builds on
32 of 38 references.
References · 32
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
- 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,407
- 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
- 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 795
- 2021DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Camerascited by 687
- 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 490
- 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 483
- 2022ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulationcited by 312
- 2025π*0.6: a VLA That Learns From Experiencecited by 303
- 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 226
- 2023Any-point Trajectory Modeling for Policy Learningcited by 215
20 more
- 2025EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Videocited by 211
- 2024EgoMimic: Scaling Imitation Learning via Egocentric Videocited by 175
- 2023HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real Worldcited by 166
- 2023Structured World Models from Human Videoscited by 147
- 2024Data Scaling Laws in Imitation Learning for Robotic Manipulationcited by 145
- 2025VGGT-SLAM: Dense RGB SLAM Optimized on the SL(4) Manifoldcited by 136
- 2025A Careful Examination of Large Behavior Models for Multitask Dexterous Manipulationcited by 129
- 2025EgoVLA: Learning Vision-Language-Action Models from Egocentric Human Videoscited by 128
- 2024Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulationcited by 124
- 2024WiLoR: End-to-end 3D Hand Localization and Reconstruction in-the-wildcited by 124
- 2025Humanoid Policy ~ Human Policycited by 78
- 2025HaWoR: World-Space Hand Motion Reconstruction from Egocentric Videoscited by 66
- 2019mm-Pose: Real-Time Human Skeletal Posture Estimation using mmWave Radars and CNNscited by 59
- 2021Human Hands as Probes for Interactive Object Understandingcited by 52
- 2023Towards Generalizable Zero-Shot Manipulation via Translating Human Interaction Planscited by 52
- 2024HRP: Human Affordances for Robotic Pre-Trainingcited by 44
- 2025DexWild: Dexterous Human Interactions for In-the-Wild Robot Policiescited by 38
- 2025ZeroMimic: Distilling Robotic Manipulation Skills from Web Videoscited by 32
- 2025EgoBridge: Domain Adaptation for Generalizable Imitation from Egocentric Human Datacited by 29
- 2023Look Ma, No Hands! Agent-Environment Factorization of Egocentric Videoscited by 21
Built on by
Citing papers per year: 2026: 2.
2 papers.