Human Demonstrations as Post-Training Data for Any Robot Body: Retargeting the Object, Not the Hand
New in Vision-language-action models; no direction yet. Vision-language-action models
Abstract (arXiv)
Human demonstrations are today used largely for pretraining vision-language-action models, with little ability to use the same data for target-embodiment or task-oriented training, which scalable deployment of robots in industrial and general environments depends on. A human demonstration recorded on a simple and scalable device such as a mobile phone can serve as post-training data for a robot whose body differs in structure and capability from the human who recorded it. This is made possible not by retargeting the hand's motion but the object's. A capture yields a body-free record: the object's pose over time, its contact intervals, and a reconstructed 3D object with provenance. Any robot, given as a URDF and a short descriptor, is solved against that record and scored in simulation against the demonstrated object motion as ground truth: the human-to-simulation leg of human-to-simulation-to-robot transfer, measured per robot. Each (robot, episode) pair gets one outcome: pass, failure with a named mode, or refusal, with its reason, before any motion. The method runs as a working system over a growing collection; the set analysed here is a sample of it: 42 captures solved on 8 robots, 157 pairs, of which 31.2% pass, 52.2% fail, 14.0% are refused and 2.5% are unauditable (kinematic feasibility, +/-50 mm / +/-15 deg, in simulation). Refusals held up: every pair the checks refuse on stored evidence yet ran anyway failed (13 of 13; 5 on a body whose public model welds its hand). A description file could not be shown to screen a robot: its outcomes were barely more predictable than guessing. No policy has yet been trained on the record. The record decides what a robot may post-train on; whether the policy then works on the real robot is the simulation-to-robot question, answered downstream.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 46 of its 58 references that are in the map. First posted on arXiv 14 Oct 2026.
Builds on
46 of 58 references.
References · 46
- 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 852
- 2024Structured 3D Latents for Scalable and Versatile 3D Generationcited by 770
- 2019MediaPipe: A Framework for Building Perception Pipelinescited by 583
- 2024Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robotscited by 394
- 2023MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrationscited by 327
- 2023Reconstructing Hands in 3D with Transformerscited by 306
- 2017Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metricscited by 264
- 2023Any-point Trajectory Modeling for Policy Learningcited by 215
- 2024EgoMimic: Scaling Imitation Learning via Egocentric Videocited by 175
- 2024DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulationcited by 160
- 2024Flow as the Cross-Domain Manipulation Interfacecited by 132
- 2024Learning Human-to-Humanoid Real-Time Whole-Body Teleoperationcited by 130
34 more
- 2024Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulationcited by 124
- 2024ExBody2: Advanced Expressive Humanoid Whole-Body Controlcited by 88
- 2024General Flow as Foundation Affordance for Scalable Robot Learningcited by 85
- 2024OKAMI: Teaching Humanoid Robots Manipulation Skills through Single Video Imitationcited by 67
- 2026Being-H0.5: Scaling Human-Centric Robot Learning for Cross-Embodiment Generalizationcited by 66
- 2024Robot See Robot Do: Imitating Articulated Object Manipulation with Monocular 4D Reconstructioncited by 54
- 2024Vision-based Manipulation from Single Human Video with Open-World Object Graphscited by 50
- 2024Robot Utility Models: General Policies for Zero-Shot Deployment in New Environmentscited by 44
- 2025Real2Render2Real: Scaling Robot Data Without Dynamics Simulation or Robot Hardwarecited by 44
- 2024SPOT: SE(3) Pose Trajectory Diffusion for Object-Centric Manipulationcited by 38
- 2025Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrationscited by 38
- 2025Crossing the Human-Robot Embodiment Gap with Sim-to-Real RL using One Human Demonstrationcited by 34
- 2024DITTO: Demonstration Imitation by Trajectory Transformationcited by 30
- 2025X-Sim: Cross-Embodiment Learning via Real-to-Sim-to-Realcited by 28
- 2025Video2Policy: Scaling up Manipulation Tasks in Simulation through Internet Videoscited by 26
- 2019Manipulation Trajectory Optimization with Online Grasp Synthesis and Selectioncited by 17
- 2021Dynamic Grasping with Reachability and Motion Awarenesscited by 16
- 2026HoMMI: Learning Whole-Body Mobile Manipulation from Human Demonstrationscited by 16
- 2025EC-Flow: Enabling Versatile Robotic Manipulation from Action-Unlabeled Videos via Embodiment-Centric Flowcited by 11
- 2026Ego2Robot: Scalable Robot Data Synthesis from Egocentric Human Datacited by 11
- 2026EgoInfinity: A Web-Scale 4D Hand-Object Interaction Data Engine for Any-View Robot Retargeting and Video-to-Action Robot Learningcited by 9
- 2026What Enables In-Context Behavior Prompting for Manipulation?cited by 7
- 2026Cloak: Zero-Shot Cross-Embodiment Manipulation by Masking the End-Effector from the VLAcited by 5
- 2026RoboPocket: Improve Robot Policies Instantly with Your Phonecited by 4
- 2026Benchmarking Vision-Language-Action Models on SO-101: Failure and Recovery Analysiscited by 4
- 2026MobileEgo Anywhere: Open Infrastructure for long horizon egocentric data on commodity hardwarecited by 3
- 2026Video2Sim2Real: Full-Stack Autonomous Dexterous Skill Acquisition from a Single Human Videocited by 3
- 2026Imitating What Works: Simulation-Filtered Modular Policy Learning from Human Videoscited by 2
- 2017Grasp that optimises objectives along post-grasp trajectoriescited by 1
- 2025Physics-Conditioned Grasping for Stable Tool Usecited by 1
- 2026FeasibleCap: Real-Time Embodiment Constraint Guidance for In-the-Wild Robot Demonstration Collectioncited by 1
- 2026Hand-centric Human-to-Robot Trajectory Transfer from Video Demonstrations via Object Category-agnostic Temporal Localizationcited by 1
- 2026PhysV2A: Reachability-Gated and Semantic-Mask-Constrained Feasibility Completion for Video-to-Robot Manipulationcited by 1
- 2026Execution-Aware Pre-Execution Ranking for Grasp-Conditioned Robotic Placementcited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2023On Hand-Held Grippers and the Morphological Gap in Human Manipulation Demonstration
- 2026Robot Learning from Human Demonstrations: Handwritten Alphabet Trajectories and Human-Likeness Evaluation
- 2026Touch2Robot: Robot Touch in the Human Demonstration Loop
- 2024Behavior Imitation for Manipulator Control and Grasping with Deep Reinforcement Learning
- 2025HRT1: One-Shot Human-to-Robot Trajectory Transfer for Mobile Manipulation
- 2026Dexterous Robot Manipulation from Human Demonstrations via Contact-Anchored Retargeting and Residual Policy Learning ↗
- 2025Imitation Learning with Precisely Labeled Human Demonstrations
- 2025RwoR: Generating Robot Demonstrations from Human Hand Collection for Policy Learning without Robot