mlsanityData to 14 Oct 2026

XSkill: Cross Embodiment Skill Discovery

arXiv 2307.09955 ↗First posted 19 Jul 2023cs.RO cs.AI cs.LGcited by 75

On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning

Abstract (arXiv)

Human demonstration videos are a widely available data source for robot learning and an intuitive user interface for expressing desired behavior. However, directly extracting reusable robot manipulation skills from unstructured human videos is challenging due to the big embodiment difference and unobserved action parameters. To bridge this embodiment gap, this paper introduces XSkill, an imitation learning framework that 1) discovers a cross-embodiment representation called skill prototypes purely from unlabeled human and robot manipulation videos, 2) transfers the skill representation to robot actions using conditional diffusion policy, and finally, 3) composes the learned skill to accomplish unseen tasks specified by a human prompt video. Our experiments in simulation and real-world environments show that the discovered skill prototypes facilitate both skill transfer and composition for unseen tasks, resulting in a more general and scalable imitation learning framework. The benchmark, code, and qualitative results are on https://xskill.cs.columbia.edu/

Key facts. Cited by 75 papers in the map (data through 14 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 63 of its 77 references that are in the map. First posted on arXiv 19 Jul 2023.

Builds on

63 of 77 references.

Closest ancestors · 11

Same direction, newest first

  1. 2023MimicPlay: Long-Horizon Imitation Learning by Watching Human Play24 Feb
  2. 2022Graph Inverse Reinforcement Learning from Diverse Videos28 Jul
  3. 2022Behavior Transformers: Cloning k modes with one stone22 Jun
  4. 2021Towards More Generalizable One-shot Visual Imitation Learning26 Oct
  5. 2021Bottom-Up Skill Discovery from Unsegmented Demonstrations for Long-Horizon Robot Manipulation28 Sep
  6. 2021XIRL: Cross-embodiment Inverse Reinforcement Learning7 Jun
  7. 2020Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations13 Mar
  8. 2019Third-Person Visual Imitation Learning via Decoupled Hierarchical Controller21 Nov
  9. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning25 Oct
  10. 2019Self-Supervised Correspondence in Visuomotor Policy Learning16 Sep
  11. 2018One-Shot Hierarchical Imitation Learning of Compound Visuomotor Tasks25 Oct

Other references · 52

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  5. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  6. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  7. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,316
  8. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  9. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,028
  10. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  11. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 725
  12. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 694
38 more
  1. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
  2. 2019Self-labelling via simultaneous clustering and representation learningcited by 528
  3. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 459
  4. 2021Implicit Behavioral Cloningcited by 452
  5. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 344
  6. 2019Learning Latent Plans from Playcited by 316
  7. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 313
  8. 2017Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperationcited by 289
  9. 2016Variational Intrinsic Controlcited by 267
  10. 2023Imitating Human Behaviour with Diffusion Modelscited by 263
  11. 2021Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasetscited by 250
  12. 2022Masked Visual Pre-training for Motor Controlcited by 248
  13. 2023Goal-Conditioned Imitation Learning using Score-based Diffusion Policiescited by 236
  14. 2017One-Shot Imitation Learningcited by 233
  15. 2017One-Shot Visual Imitation Learning via Meta-Learningcited by 228
  16. 2020Accelerating Reinforcement Learning with Learned Skill Priorscited by 223
  17. 2019Efficient Exploration via State Marginal Matchingcited by 188
  18. 2021Reinforcement Learning with Prototypical Representationscited by 179
  19. 2017Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translationcited by 167
  20. 2022Human-to-Robot Imitation in the Wildcited by 161
  21. 2018One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learningcited by 148
  22. 2020Parrot: Data-Driven Behavioral Priors for Reinforcement Learningcited by 138
  23. 2018Variational Option Discovery Algorithmscited by 122
  24. 2021Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videoscited by 119
  25. 2021APS: Active Pretraining with Successor Featurescited by 112
  26. 2019AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videoscited by 105
  27. 2017Vision-Based Multi-Task Manipulation for Inexpensive Robots Using End-To-End Learning from Demonstrationcited by 100
  28. 2018Hierarchical Imitation and Reinforcement Learningcited by 88
  29. 2021Learning by Watching: Physical Imitation of Manipulation Skills from Human Videoscited by 87
  30. 2019IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Datacited by 86
  31. 2020Reinforcement Learning with Videos: Combining Offline Observations with Interactioncited by 82
  32. 2020Transformers for One-Shot Visual Imitationcited by 78
  33. 2016Unsupervised Perceptual Rewards for Imitation Learningcited by 65
  34. 2018TACO: Learning Task Decomposition via Temporal Alignment for Controlcited by 62
  35. 2020Learning Robot Skills with Temporal Variational Inferencecited by 54
  36. 2017Multi-Level Discovery of Deep Optionscited by 51
  37. 2019Graph-Structured Visual Imitationcited by 43
  38. 2016CliqueCNN: Deep Unsupervised Exemplar Learningcited by 41

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 25; 2025: 33; 2026: 16.

75 papers.

Sort:

By area:

  1. 2024RoboMIND: Benchmark on Multi-embodiment Intelligence Normative Data for Robot Manipulationcited by 175
  2. 2024DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulationcited by 160
  3. 2024Flow as the Cross-Domain Manipulation Interfacecited by 132
  4. 2025AdaWorld: Learning Adaptable World Models with Latent Actionscited by 114
  5. 2025Humanoid Policy ~ Human Policycited by 78
  6. 2025Motion Tracks: A Unified Representation for Human-Robot Transfer in Few-Shot Imitation Learningcited by 59
  7. 2024Vid2Robot: End-to-end Video-conditioned Policy Learning with Cross-Attention Transformerscited by 52
  8. 2024Vision-based Manipulation from Single Human Video with Open-World Object Graphscited by 50
  9. 2025NavDP: Learning Sim-to-Real Navigation Diffusion Policy with Privileged Information Guidancecited by 49
  10. 2025VidBot: Learning Generalizable 3D Actions from In-the-Wild 2D Human Videos for Zero-Shot Robotic Manipulationcited by 46
  11. 2025Diffusion Models for Robotic Manipulation: A Surveycited by 41
  12. 2025Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrationscited by 38
38 more
  1. 2025DexMachina: Functional Retargeting for Bimanual Dexterous Manipulationcited by 37
  2. 2025UniSkill: Imitating Human Videos via Cross-Embodiment Skill Representationscited by 35
  3. 2023MimicTouch: Leveraging Multi-modal Human Tactile Demonstrations for Contact-rich Manipulationcited by 34
  4. 2025Compliant Residual DAgger: Improving Real-World Contact-Rich Manipulation with Human Correctionscited by 29
  5. 2025X-Sim: Cross-Embodiment Learning via Real-to-Sim-to-Realcited by 28
  6. 2026SkillFlow:Benchmarking Lifelong Skill Discovery and Evolution for Autonomous Agentscited by 25
  7. 2025TASTE-Rob: Advancing Video Generation of Task-Oriented Hand-Object Interaction for Generalizable Robotic Manipulationcited by 24
  8. 2024Don't Start from Scratch: Behavioral Refinement via Interpolant-based Policy Diffusioncited by 21
  9. 2025Dexterous Manipulation through Imitation Learning: A Surveycited by 21
  10. 2025Robot Learning from a Physical World Modelcited by 20
  11. 2024Generative Image as Action Modelscited by 18
  12. 2026LAP: Language-Action Pre-Training Enables Zero-shot Cross-Embodiment Transfercited by 18
  13. 2024Inference via Interpolation: Contrastive Representations Provably Enable Planning and Inferencecited by 17
  14. 2024Diffusion Model for Planning: A Systematic Literature Reviewcited by 17
  15. 2024A Comprehensive Survey of Cross-Domain Policy Transfer for Embodied Agentscited by 16
  16. 2024Language-Driven 6-DoF Grasp Detection Using Negative Prompt Guidancecited by 16
  17. 2024The One RING: a Robotic Indoor Navigation Generalistcited by 16
  18. 2026HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videoscited by 16
  19. 2024One-Shot Imitation under Mismatched Executioncited by 12
  20. 2026VideoWorld 2: Learning Transferable Knowledge from Real-world Videoscited by 12
  21. 2024Human-Agent Joint Learning for Efficient Robot Manipulation Skill Acquisitioncited by 10
  22. 2025From Screens to Scenes: A Survey of Embodied AI in Healthcarecited by 10
  23. 2025MoE-DP: An MoE-Enhanced Diffusion Policy for Robust Long-Horizon Robotic Manipulation with Skill Decomposition and Failure Recoverycited by 10
  24. 2024PEAC: Unsupervised Pre-training for Cross-Embodiment Reinforcement Learningcited by 9
  25. 2024xTED: Cross-Domain Adaptation via Diffusion-Based Trajectory Editingcited by 9
  26. 2025SPECI: Skill Prompts based Hierarchical Continual Imitation Learning for Robot Manipulationcited by 9
  27. 2025Demystifying Diffusion Policies: Action Memorization and Simple Lookup Table Alternativescited by 9
  28. 2024Generalize by Touching: Tactile Ensemble Skill Transfer for Robotic Furniture Assemblycited by 7
  29. 2025BOSS: Benchmark for Observation Space Shift in Long-Horizon Taskcited by 7
  30. 2025Chain-of-Modality: Learning Manipulation Programs from Multimodal Human Videos with Vision-Language-Modelscited by 7
  31. 2025Multi-Loco: Unifying Multi-Embodiment Legged Locomotion via Reinforcement Learning Augmented Diffusioncited by 7
  32. 2024Polaris: Open-ended Interactive Robotic Manipulation via Syn2Real Visual Grounding and Large Language Modelscited by 6
  33. 2025What Changed and What Could Have Changed? State-Change Counterfactuals for Procedure-Aware Video Representation Learningcited by 6
  34. 2025X-Diffusion: Training Diffusion Policies on Cross-Embodiment Human Demonstrationscited by 6
  35. 2026Breaking Lock-In: Preserving Steerability under Low-Data VLA Post-Trainingcited by 6
  36. 2025LodeStar: Long-horizon Dexterity via Synthetic Data Augmentation from Human Demonstrationscited by 4
  37. 2024InfoCon: Concept Discovery with Generative and Discriminative Informativenesscited by 3
  38. 2025Traj2Action: A Co-Denoising Framework for Trajectory-Guided Human-to-Robot Skill Transfercited by 3

50 most cited of 75 All 75 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025ZeroMimic: Distilling Robotic Manipulation Skills from Web Videos
  2. 2026Human-to-Robot Interaction: Learning from Video Demonstration for Robot Imitation
  3. 2026Bridging the Embodiment Gap: Disentangled Cross-Embodiment Video Editing ↗
  4. 2026Cross-Embodiment Transfer via Behavior-Aligned Representations
  5. 2022Eliciting Compatible Demonstrations for Multi-Human Imitation Learning
  6. 2025Crossing the Human-Robot Embodiment Gap with Sim-to-Real RL using One Human Demonstration
  7. 2025RoboManipBaselines: A Unified Framework for Imitation Learning in Robotic Manipulation across Real and Simulation Environments
  8. 2025SKIL: Semantic Keypoint Imitation Learning for Generalizable Data-efficient Manipulation