mlsanityData to 14 Oct 2026

Crossing the Human-Robot Embodiment Gap with Sim-to-Real RL using One Human Demonstration

arXiv 2504.12609 ↗First posted 17 Apr 2025cs.RO cs.AIcited by 34

Not on the map. Why?

Abstract (arXiv)

Teaching robots dexterous manipulation skills often requires collecting hundreds of demonstrations using wearables or teleoperation, a process that is challenging to scale. Videos of human-object interactions are easier to collect and scale, but leveraging them directly for robot learning is difficult due to the lack of explicit action labels and human-robot embodiment differences. We propose Human2Sim2Robot, a novel real-to-sim-to-real framework for training dexterous manipulation policies using only one RGB-D video of a human demonstrating a task. Our method utilizes reinforcement learning (RL) in simulation to cross the embodiment gap without relying on wearables, teleoperation, or large-scale data collection. From the video, we extract: (1) the object pose trajectory to define an object-centric, embodiment-agnostic reward, and (2) the pre-manipulation hand pose to initialize and guide exploration during RL training. These components enable effective policy learning without any task-specific reward tuning. In the single human demo regime, Human2Sim2Robot outperforms object-aware replay by over 55% and imitation learning by over 68% on grasping, non-prehensile manipulation, and multi-step tasks. Website: https://human2sim2robot.github.io

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Builds on 48 of its 59 references that are in the map. First posted on arXiv 17 Apr 2025.

Builds on

48 of 59 references.

References · 48

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  3. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  4. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  5. 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
  6. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  7. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  8. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 998
  9. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 862
  10. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  11. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 523
  12. 2023FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objectscited by 429
36 more
  1. 2024Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robotscited by 394
  2. 2022Learning robust perceptive locomotion for quadrupedal robots in the wildcited by 382
  3. 2024Latent Action Pretraining from Videoscited by 321
  4. 2023Reconstructing Hands in 3D with Transformerscited by 306
  5. 2017Asymmetric Actor Critic for Image-Based Robot Learningcited by 238
  6. 2021DexMV: Imitation Learning for Dexterous Manipulation from Human Videoscited by 206
  7. 2022Rapid Locomotion via Reinforcement Learningcited by 168
  8. 2023Extreme Parkour with Legged Robotscited by 164
  9. 2024DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulationcited by 160
  10. 2024Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulationcited by 144
  11. 2024ALOHA Unleashed: A Simple Recipe for Robot Dexteritycited by 134
  12. 2023Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learningcited by 133
  13. 2023UniDexGrasp: Universal Robotic Dexterous Grasping via Learning Diverse Proposal Generation and Goal-Conditioned Policycited by 124
  14. 2022Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learningcited by 119
  15. 2022In-Hand Object Rotation via Rapid Motor Adaptationcited by 118
  16. 2020The Ingredients of Real-World Robotic Reinforcement Learningcited by 117
  17. 2024DexMimicGen: Automated Data Generation for Bimanual Dexterous Manipulation via Imitation Learningcited by 116
  18. 2021XIRL: Cross-embodiment Inverse Reinforcement Learningcited by 107
  19. 2023UniDexGrasp++: Improving Dexterous Grasping Policy Learning via Geometry-aware Curriculum and Iterative Generalist-Specialist Learningcited by 107
  20. 2023LEAP Hand: Low-Cost, Efficient, and Anthropomorphic Hand for Robot Learningcited by 79
  21. 2024OKAMI: Teaching Humanoid Robots Manipulation Skills through Single Video Imitationcited by 67
  22. 2024Robot See Robot Do: Imitating Articulated Object Manipulation with Monocular 4D Reconstructioncited by 54
  23. 2022Graph Inverse Reinforcement Learning from Diverse Videoscited by 46
  24. 2024Omnigrasp: Grasping Diverse Objects with Simulated Humanoidscited by 46
  25. 2024DextrAH-G: Pixels-to-Action Dexterous Arm-Hand Grasping with Geometric Fabricscited by 42
  26. 2024Object-Centric Dexterous Manipulation from Human Motion Datacited by 41
  27. 2024Hand-Object Interaction Pretraining from Videoscited by 39
  28. 2022Demonstrate Once, Imitate Immediately (DOME): Learning Visual Servoing for One-Shot Imitation Learningcited by 38
  29. 2022Learning Continuous Grasping Function with a Dexterous Hand from Human Demonstrationscited by 34
  30. 2024ARCap: Collecting High-quality Human Demonstrations for Robot Learning with Augmented Reality Feedbackcited by 31
  31. 2024DITTO: Demonstration Imitation by Trajectory Transformationcited by 30
  32. 2022Learning Dexterous Manipulation from Exemplar Object Trajectories and Pre-Graspscited by 29
  33. 2024Bridging the Human to Robot Dexterity Gap through Object-Oriented Rewardscited by 28
  34. 2025DexForce: Extracting Force-informed Actions from Kinesthetic Demonstrations for Dexterous Manipulationcited by 24
  35. 2023One-Shot Imitation Learning: A Pose Estimation Perspectivecited by 21
  36. 2024A Comparison of Imitation Learning Algorithms for Bimanual Manipulationcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 24.

34 papers.

Sort:

By area:

  1. 2025SPIDER: Scalable Physics-Informed Dexterous Retargetingcited by 32
  2. 2025X-Sim: Cross-Embodiment Learning via Real-to-Sim-to-Realcited by 28
  3. 2025Robot Learning from a Physical World Modelcited by 20
  4. 2025DemoDiffusion: One-Shot Human Imitation using pre-trained Diffusion Policycited by 19
  5. 2026LAP: Language-Action Pre-Training Enables Zero-shot Cross-Embodiment Transfercited by 18
  6. 2026SimToolReal: An Object-Centric Policy for Zero-Shot Dexterous Tool Manipulationcited by 15
  7. 2026Do as I Do: Dexterous Manipulation Data from Everyday Human Videoscited by 13
  8. 2026Dexterous Manipulation Policies from RGB Human Videos via 3D Hand-Object Trajectory Reconstructioncited by 11
  9. 2025DexMan: Learning Bimanual Dexterous Manipulation from Human and Generated Videoscited by 10
  10. 2025Scaffolding Dexterous Manipulation with Vision-Language Modelscited by 9
  11. 2026FlowHOI: Flow-based Semantics-Grounded Generation of Hand-Object Interactions for Dexterous Robot Manipulationcited by 7
  12. 2025X-Diffusion: Training Diffusion Policies on Cross-Embodiment Human Demonstrationscited by 6
22 more
  1. 2026Learning Dexterous Manipulation Using Contact Wrench Guidance From Human Demonstrationcited by 6
  2. 2026Robot Learning from Human Videos: A Surveycited by 5
  3. 2026LUCID: Learning Embodiment-Agnostic Intent Models from Unstructured Human Videos for Scalable Dexterous Robot Skill Acquisitioncited by 5
  4. 2026CRAFT: Video Diffusion for Bimanual Robot Data Generationcited by 4
  5. 2026A Minimalist Retargeting-Guided Reinforcement Learning Recipe for Dexterous Manipulationcited by 4
  6. 2026D-REX: Differentiable Real-to-Sim-to-Real Engine for Learning Dexterous Graspingcited by 3
  7. 2025Joint Flow Trajectory Optimization For Feasible Robot Motion Generation from Video Demonstrationscited by 2
  8. 2025DeLTa: Demonstration and Language-Guided Novel Transparent Object Manipulationcited by 2
  9. 2026Imitating What Works: Simulation-Filtered Modular Policy Learning from Human Videoscited by 2
  10. 2026Scaling Sim-to-Real VLA Reinforcement Learning with Generative 3D Worldscited by 2
  11. 2025UniBYD: A Unified Framework for Learning Robotic Manipulation Across Embodiments Beyond Imitation of Human Demonstrationscited by 1
  12. 2026Play2Perfect: What Matters in Dexterous Play Pretraining for Precise Assembly?cited by 1
  13. 2026DexMani: Human-Derived Manipulability Guidance for Dexterous Rotationcited by 1
  14. 2026The Embodiment Gap in Robot Foundation Modelscited by 1
  15. 2026KnowDemo: Knowledge-Guided Robot Demonstration Generation from Human Videoscited by 1
  16. 2026DexAgent: An Agentic Human2Sim2Robot Framework for Dexterous Manipulation with Self-Evolving Tool Librarycited by 1
  17. 2026HandelBot: Real-World Piano Playing via Fast Adaptation of Dexterous Robot Policiescited by 0
  18. 2026Scalable Multi-Task Data Generation via Reinforcement Learning for Language-Conditioned Bimanual Dexterous Manipulationcited by 0
  19. 2026Grasp2Twist: Learning Bimanual Dexterous Jar Opening by Reinforcement Learningcited by 0
  20. 2026X-Reset: Scaling Object-Centric Reinforcement Learning via Cross-Embodiment Resetscited by 0
  21. 2026GOTT: Object-centric Dexterous Manipulation with a Reusable Cross-Embodiment Primitivecited by 0
  22. 2026Human Demonstrations as Post-Training Data for Any Robot Body: Retargeting the Object, Not the Handcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Dex-One2Many: Learning Dexterous Manipulation from a Single Human Demonstration
  2. 2026Dexterous Point Policy: Learning Point-based Dexterous Hand Policies from Human Demonstrations
  3. 2024Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning
  4. 2026EaDex: A Cross-Embodiment Dexterous Manipulation Framework from Low-Cost Demonstrations
  5. 2026DemoBot: Efficient Learning of Bimanual Manipulation with Dexterous Hands From Third-Person Human Videos
  6. 2025DexterityGen: Foundation Controller for Unprecedented Dexterity
  7. 2026UniDex-ViTac: Learning Unified Visuo-Tactile Dexterous Manipulation Policy from Human Video Data ↗
  8. 2023Learning a Universal Human Prior for Dexterous Manipulation from Human Preference