# UniSkill: Imitating Human Videos via Cross-Embodiment Skill Representations

arXiv: https://arxiv.org/abs/2505.08787 · HTML: https://mlsanity.com/paper/2505.08787/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 31 references that are in the map.
- First posted on arXiv 13 May 2025.

## Builds on (most cited first, 30 of 31)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [Depth Anything V2](https://mlsanity.com/paper/2406.09414/) (2024)
- [LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning](https://mlsanity.com/paper/2306.03310/) (2023)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset](https://mlsanity.com/paper/2403.12945/) (2024)
- [Genie: Generative Interactive Environments](https://mlsanity.com/paper/2402.15391/) (2024)
- [What Matters in Learning from Offline Human Demonstrations for Robot Manipulation](https://mlsanity.com/paper/2108.03298/) (2021)
- [BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning](https://mlsanity.com/paper/2202.02005/) (2022)
- [Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation](https://mlsanity.com/paper/2401.02117/) (2024)
- [Latent Action Pretraining from Videos](https://mlsanity.com/paper/2410.11758/) (2024)
- [H2O: Two Hands Manipulating Objects for First Person Interaction Recognition](https://mlsanity.com/paper/2104.11181/) (2021)
- [Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasets](https://mlsanity.com/paper/2109.13396/) (2021)
- [MimicPlay: Long-Horizon Imitation Learning by Watching Human Play](https://mlsanity.com/paper/2302.12422/) (2023)
- [Any-point Trajectory Modeling for Policy Learning](https://mlsanity.com/paper/2401.00025/) (2023)
- [Spatial-Temporal Transformer Networks for Traffic Flow Forecasting](https://mlsanity.com/paper/2001.02908/) (2020)
- [EgoMimic: Scaling Imitation Learning via Egocentric Video](https://mlsanity.com/paper/2410.24221/) (2024)
- [One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learning](https://mlsanity.com/paper/1802.01557/) (2018)
- [Flow as the Cross-Domain Manipulation Interface](https://mlsanity.com/paper/2407.15208/) (2024)
- [Learning to Act without Actions](https://mlsanity.com/paper/2312.10812/) (2023)
- [XSkill: Cross Embodiment Skill Discovery](https://mlsanity.com/paper/2307.09955/) (2023)
- [Motion Tracks: A Unified Representation for Human-Robot Transfer in Few-Shot Imitation Learning](https://mlsanity.com/paper/2501.06994/) (2025)
- [Vid2Robot: End-to-end Video-conditioned Policy Learning with Cross-Attention Transformers](https://mlsanity.com/paper/2403.12943/) (2024)
- [Cross-Domain Transfer via Semantic Skill Imitation](https://mlsanity.com/paper/2212.07407/) (2022)

## Built on by (most cited first, 30 of 35)

- [π0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilities](https://mlsanity.com/paper/2604.15483/) (2026)
- [Vision-Language-Action Models for Robotics: A Review Towards Real-World Applications](https://mlsanity.com/paper/2510.07077/) (2025)
- [Robot Learning from a Physical World Model](https://mlsanity.com/paper/2511.07416/) (2025)
- [HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videos](https://mlsanity.com/paper/2605.24934/) (2026)
- [TraceGen: World Modeling in 3D Trace Space Enables Learning from Cross-Embodiment Videos](https://mlsanity.com/paper/2511.21690/) (2025)
- [ConLA: Contrastive Latent Action Learning from Human Videos for Robotic Manipulation](https://mlsanity.com/paper/2602.00557/) (2026)
- [MVP-LAM: Learning Action-Centric Latent Action via Cross-Viewpoint Reconstruction](https://mlsanity.com/paper/2602.03668/) (2026)
- [Co-Evolving Latent Action World Models](https://mlsanity.com/paper/2510.26433/) (2025)
- [Developing Vision-Language-Action Model from Egocentric Videos](https://mlsanity.com/paper/2509.21986/) (2025)
- [OSMO: Open-Source Tactile Glove for Human-to-Robot Skill Transfer](https://mlsanity.com/paper/2512.08920/) (2025)
- [Olaf-World: Orienting Latent Actions for Video World Modeling](https://mlsanity.com/paper/2602.10104/) (2026)
- [DiLA: Disentangled Latent Action World Models](https://mlsanity.com/paper/2605.15725/) (2026)
- [Cross-Hand Latent Representation for Vision-Language-Action Models](https://mlsanity.com/paper/2603.10158/) (2026)
- [Hierarchical Latent Action Model](https://mlsanity.com/paper/2603.05815/) (2026)
- [UniLACT: Depth-Aware RGB Latent Action Learning for Vision-Language-Action Models](https://mlsanity.com/paper/2602.20231/) (2026)
- [Robot Learning from Human Videos: A Survey](https://mlsanity.com/paper/2604.27621/) (2026)
- [HARP-VLA: Human-Robot Aligned Representation Learning for Vision-Language-Action Model](https://mlsanity.com/paper/2605.31234/) (2026)
- [Human-Centric Transferable Tactile Pre-Training for Dexterous Robotic Manipulation](https://mlsanity.com/paper/2607.01067/) (2026)
- [Learning Skills from Action-Free Videos](https://mlsanity.com/paper/2512.20052/) (2025)
- [SeeTraceAct: Visibility-Aware Latent Planning from Cross-Embodiment Demonstration Videos](https://mlsanity.com/paper/2606.02745/) (2026)
- [Imitation from Heterogeneous Demonstrations using Grounded Latent-Action World Models](https://mlsanity.com/paper/2606.21672/) (2026)
- [ShadowDancer: Teaching Video World Models Any Action by Learning Unified Dynamics Representations from a Video and Its Shadow](https://mlsanity.com/paper/2607.28362/) (2026)
- [The Imitator Game: Benchmarking Robot Imitative Ability Beyond Action Prediction](https://mlsanity.com/paper/2608.22301/) (2026)
- [PonderPounce: A Pretrained MLLM as an Episode Context Engine for Robot Control](https://mlsanity.com/paper/2608.24115/) (2026)
- [In-Context Learning for Robots: Methods and Applications](https://mlsanity.com/paper/2609.36012/) (2026)
- [EgoRecovery: Acquiring Failure Recovery Ability Through Human Recovery Demonstration](https://mlsanity.com/paper/2607.19745/) (2026)
- [BooST: Bridging Semantics and Motions for Efficient Skill Transfer](https://mlsanity.com/paper/2608.10600/) (2026)
- [ZETA: A Controlled Study of Zero-Shot Cross-Embodiment VLA Transfer for Tabletop Manipulation](https://mlsanity.com/paper/2609.02546/) (2026)
- [Toward Unified Robot Learning: Bridging Representation, Vision-Language-Action, and World Models](https://mlsanity.com/paper/2609.03927/) (2026)
- [Reconstructing Is Not Acting: Action-Centric Latent Dynamics Modeling](https://mlsanity.com/paper/2609.15189/) (2026)

## Abstract (arXiv)

Mimicry is a fundamental learning mechanism in humans, enabling individuals to learn new tasks by observing and imitating experts. However, applying this ability to robots presents significant challenges due to the inherent differences between human and robot embodiments in both their visual appearance and physical capabilities. While previous methods bridge this gap using cross-embodiment datasets with shared scenes and tasks, collecting such aligned data between humans and robots at scale is not trivial. In this paper, we propose UniSkill, a novel framework that learns embodiment-agnostic skill representations from large-scale cross-embodiment video data without any labels, enabling skills extracted from human video prompts to effectively transfer to robot policies trained only on robot data. Our experiments in both simulation and real-world environments show that our cross-embodiment skills successfully guide robots in selecting appropriate actions, even with unseen video prompts. The project website can be found at: https://kimhanjung.github.io/UniSkill.