mlsanityData to 14 Oct 2026

X-Sim: Cross-Embodiment Learning via Real-to-Sim-to-Real

arXiv 2505.07096 ↗First posted 11 May 2025cs.RO cs.AI cs.LGcited by 28

On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning

Abstract (arXiv)

Human videos offer a scalable way to train robot manipulation policies, but lack the action labels needed by standard imitation learning algorithms. Existing cross-embodiment approaches try to map human motion to robot actions, but often fail when the embodiments differ significantly. We propose X-Sim, a real-to-sim-to-real framework that uses object motion as a dense and transferable signal for learning robot policies. X-Sim starts by reconstructing a photorealistic simulation from an RGBD human video and tracking object trajectories to define object-centric rewards. These rewards are used to train a reinforcement learning (RL) policy in simulation. The learned policy is then distilled into an image-conditioned diffusion policy using synthetic rollouts rendered with varied viewpoints and lighting. To transfer to the real world, X-Sim introduces an online domain adaptation technique that aligns real and simulated observations during deployment. Importantly, X-Sim does not require any robot teleoperation data. We evaluate it across 5 manipulation tasks in 2 environments and show that it: (1) improves task progress by 30% on average over hand-tracking and sim-to-real baselines, (2) matches behavior cloning with 10x less data collection time, and (3) generalizes to new camera viewpoints and test-time changes. Code and videos are available at https://portal-cornell.github.io/X-Sim/.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 55 of its 63 references that are in the map. First posted on arXiv 11 May 2025.

Builds on

55 of 63 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025Motion Tracks: A Unified Representation for Human-Robot Transfer in Few-Shot Imitation Learning13 Jan
  2. 2024ALOHA Unleashed: A Simple Recipe for Robot Dexterity17 Oct
  3. 2024Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulation6 Mar
  4. 2023One-Shot Imitation Learning: A Pose Estimation Perspective18 Oct
  5. 2023GELLO: A General, Low-Cost, and Intuitive Teleoperation Framework for Robot Manipulators22 Sep
  6. 2023XSkill: Cross Embodiment Skill Discovery19 Jul
  7. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware23 Apr
  8. 2023MimicPlay: Long-Horizon Imitation Learning by Watching Human Play24 Feb
  9. 2022Graph Inverse Reinforcement Learning from Diverse Videos28 Jul
  10. 2022Dexterous Imitation Made Easy: A Learning-Based Framework for Efficient Dexterous Manipulation24 Mar
  11. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning4 Feb
  12. 2021XIRL: Cross-embodiment Inverse Reinforcement Learning7 Jun

Other references · 43

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2023Segment Anythingcited by 8,234
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  4. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  5. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  6. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,761
  7. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
  8. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,225
  9. 20242D Gaussian Splatting for Geometrically Accurate Radiance Fieldscited by 1,076
  10. 2015End-to-End Training of Deep Visuomotor Policiescited by 875
  11. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 862
  12. 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 852
31 more
  1. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 672
  2. 2024InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Modelscited by 433
  3. 2023FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objectscited by 429
  4. 2024Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robotscited by 394
  5. 2023Reconstructing Hands in 3D with Transformerscited by 306
  6. 2018Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experiencecited by 217
  7. 2022Human-to-Robot Imitation in the Wildcited by 161
  8. 2024Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulationcited by 153
  9. 2022VideoDex: Learning Dexterity from Internet Videoscited by 106
  10. 2024Automated Creation of Digital Cousins for Robust Policy Learningcited by 91
  11. 2024From Imitation to Refinement -- Residual RL for Precise Assemblycited by 90
  12. 2025St4RTrack: Simultaneous 4D Reconstruction and Tracking in the Worldcited by 89
  13. 2022Robotic Telekinesis: Learning a Robotic Hand Imitator by Watching Humans on Youtubecited by 82
  14. 2024OPEN TEACH: A Versatile Teleoperation System for Robotic Manipulationcited by 71
  15. 2025Phantom: Training Robots Without Robots Using Only Human Videoscited by 70
  16. 2024OKAMI: Teaching Humanoid Robots Manipulation Skills through Single Video Imitationcited by 67
  17. 2024Robot See Robot Do: Imitating Articulated Object Manipulation with Monocular 4D Reconstructioncited by 54
  18. 2023Towards Generalizable Zero-Shot Manipulation via Translating Human Interaction Planscited by 52
  19. 2024Vid2Robot: End-to-end Video-conditioned Policy Learning with Cross-Attention Transformerscited by 52
  20. 2023Zero-Shot Robot Manipulation from Passive Human Videoscited by 51
  21. 2024Vision-based Manipulation from Single Human Video with Open-World Object Graphscited by 50
  22. 2023Ditto in the House: Building Articulation Models of Indoor Scenes through Interactive Perceptioncited by 41
  23. 2022Learning Continuous Grasping Function with a Dexterous Hand from Human Demonstrationscited by 34
  24. 2022Learning to Imitate Object Interactions from Internet Videoscited by 34
  25. 2025Crossing the Human-Robot Embodiment Gap with Sim-to-Real RL using One Human Demonstrationcited by 34
  26. 2025DRAWER: Digital Reconstruction and Articulation With Environment Realismcited by 29
  27. 2024Bridging the Human to Robot Dexterity Gap through Object-Oriented Rewardscited by 28
  28. 2025Video2Policy: Scaling up Manipulation Tasks in Simulation through Internet Videoscited by 26
  29. 2020Sim2Real2Sim: Bridging the Gap Between Simulation and Real-World in Flexible Object Manipulationcited by 14
  30. 2025Shadow: Leveraging Segmentation Masks for Cross-Embodiment Policy Transfercited by 14
  31. 2024One-Shot Imitation under Mismatched Executioncited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 20.

28 papers.

Sort:

By area:

  1. 2025PolaRiS: Scalable Real-to-Sim Evaluations for Generalist Robot Policiescited by 33
  2. 2025The Reality Gap in Robotics: Challenges, Solutions, and Best Practicescited by 24
  3. 2025Robot Learning from a Physical World Modelcited by 20
  4. 2025StereoVLA: Enhancing Vision-Language-Action Models with Stereo Visioncited by 18
  5. 2026SimToolReal: An Object-Centric Policy for Zero-Shot Dexterous Tool Manipulationcited by 15
  6. 2025Generalizable Domain Adaptation for Sim-and-Real Policy Co-Trainingcited by 10
  7. 2025OXE-AugE: A Large-Scale Robot Augmentation of OXE for Scaling Cross-Embodiment Policy Learningcited by 10
  8. 2026Cross-Hand Latent Representation for Vision-Language-Action Modelscited by 7
  9. 2026Reconstruction by Generation: 3D Multi-Object Scene Reconstruction from Sparse Observationscited by 7
  10. 2025X-Diffusion: Training Diffusion Policies on Cross-Embodiment Human Demonstrationscited by 6
  11. 2026Robot Learning from Human Videos: A Surveycited by 5
  12. 2026Cloak: Zero-Shot Cross-Embodiment Manipulation by Masking the End-Effector from the VLAcited by 5
16 more
  1. 2026CRAFT: Video Diffusion for Bimanual Robot Data Generationcited by 4
  2. 2026Scaling Sim-to-Real VLA Reinforcement Learning with Generative 3D Worldscited by 2
  3. 2025RobotSeg: A Model and Dataset for Segmenting Robots in Image and Videocited by 1
  4. 2026CoEnv: Driving Embodied Multi-Agent Collaboration via Compositional Environmentcited by 1
  5. 2026EgoFun3D: Modeling Interactive Objects from Egocentric Videos using Function Templatescited by 1
  6. 2026From Seeing to Simulating: Generative High-Fidelity Simulation with Digital Cousins for Generalizable Robot Learning and Evaluationcited by 1
  7. 2026GASE: Gaussian Splatting-Based Automated System for Reconstructing Embodied-Simulation Environmentscited by 1
  8. 2026Support-Constrained RL Enables Real-World Policy Improvement without Real-World Experiencecited by 1
  9. 2026OmniTacTune: Policy-Agnostic Real-World RL for Tactile Residual Adaptation of Visual Policiescited by 1
  10. 2026PRISM: Personalized Robotic Dataset Generation via Image-based Scene and Motion Synthesiscited by 1
  11. 2026The Imitator Game: Benchmarking Robot Imitative Ability Beyond Action Predictioncited by 1
  12. 2026KnowDemo: Knowledge-Guided Robot Demonstration Generation from Human Videoscited by 1
  13. 2026Scalable Multi-Task Data Generation via Reinforcement Learning for Language-Conditioned Bimanual Dexterous Manipulationcited by 0
  14. 2026CAPEX: Efficiently Distilling Foundation Model Behavior into Deployable Robot Policies through Experience-Adaptive Reasoningcited by 0
  15. 2026X-Reset: Scaling Object-Centric Reinforcement Learning via Cross-Embodiment Resetscited by 0
  16. 2026Human Demonstrations as Post-Training Data for Any Robot Body: Retargeting the Object, Not the Handcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Efficient Sim-to-Real Transfer of World-Action Models from Synthetic Priors
  2. 2026CLAP: Cross-Embodiment Video World Models are Zero-Shot Physical Simulators
  3. 2025AnchorDream: Repurposing Video Diffusion for Embodiment-Aware Robot Data Synthesis
  4. 2025SimLauncher: Launching Sample-Efficient Real-world Robotic Reinforcement Learning via Simulation Pre-training
  5. 2026Grounded-Dreamer: Robot Learning from World Model Synthetic Data with Physical Grounding
  6. 2026XEWorld: Can Action-Conditioned World Models Generalize to Unseen Robot Embodiments?
  7. 2024Representation Alignment from Human Feedback for Cross-Embodiment Reward Learning from Mixed-Quality Demonstrations
  8. 2026Grounding Sim-to-Real Generalization in Robotic Manipulation: An Empirical Study with Vision-Language-Action Models