# Cross-Embodiment Robot Manipulation Skill Transfer using Latent Space Alignment

arXiv: https://arxiv.org/abs/2406.01968 · HTML: https://mlsanity.com/paper/2406.01968/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 44 references that are in the map.
- First posted on arXiv 4 Jun 2024.

## Builds on (most cited first, 30 of 33)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [R3M: A Universal Visual Representation for Robot Manipulation](https://mlsanity.com/paper/2203.12601/) (2022)
- [robosuite: A Modular Simulation Framework and Benchmark for Robot Learning](https://mlsanity.com/paper/2009.12293/) (2020)
- [Model-Based Reinforcement Learning for Atari](https://mlsanity.com/paper/1903.00374/) (2019)
- [Time-Contrastive Networks: Self-Supervised Learning from Video](https://mlsanity.com/paper/1704.06888/) (2017)
- [Learning Invariant Representations for Reinforcement Learning without Reconstruction](https://mlsanity.com/paper/2006.10742/) (2020)
- [Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation](https://mlsanity.com/paper/1710.04615/) (2017)
- [Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images](https://mlsanity.com/paper/1506.07365/) (2015)
- [Contrastive Learning as Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2206.07568/) (2022)
- [Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model](https://mlsanity.com/paper/1907.00953/) (2019)
- [The Surprising Effectiveness of Representation Learning for Visual Imitation](https://mlsanity.com/paper/2112.01511/) (2021)
- [Unsupervised State Representation Learning in Atari](https://mlsanity.com/paper/1906.08226/) (2019)
- [Self-Supervised Policy Adaptation during Deployment](https://mlsanity.com/paper/2007.04309/) (2020)
- [Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation](https://mlsanity.com/paper/2109.01115/) (2021)
- [Self-Supervised Correspondence in Visuomotor Policy Learning](https://mlsanity.com/paper/1909.06933/) (2019)
- [XIRL: Cross-embodiment Inverse Reinforcement Learning](https://mlsanity.com/paper/2106.03911/) (2021)
- [Third-Person Imitation Learning](https://mlsanity.com/paper/1703.01703/) (2017)
- [Bootstrap Latent-Predictive Representations for Multitask Reinforcement Learning](https://mlsanity.com/paper/2004.14646/) (2020)
- [RL-CycleGAN: Reinforcement Learning Aware Simulation-To-Real](https://mlsanity.com/paper/2006.09001/) (2020)
- [Learning Invariant Feature Spaces to Transfer Skills with Reinforcement Learning](https://mlsanity.com/paper/1703.02949/) (2017)
- [Learning Plannable Representations with Causal InfoGAN](https://mlsanity.com/paper/1807.09341/) (2018)
- [Model-Based Inverse Reinforcement Learning from Visual Demonstrations](https://mlsanity.com/paper/2010.09034/) (2020)
- [Learning Cross-Domain Correspondence for Control with Dynamics Cycle-Consistency](https://mlsanity.com/paper/2012.09811/) (2020)
- [Learning Robotic Manipulation through Visual Planning and Acting](https://mlsanity.com/paper/1905.04411/) (2019)
- [Hierarchically Decoupled Imitation for Morphological Transfer](https://mlsanity.com/paper/2003.01709/) (2020)

## Built on by (most cited first, 18 of 18)

- [Cross-Embodiment Dexterous Grasping with Reinforcement Learning](https://mlsanity.com/paper/2410.02479/) (2024)
- [LAP: Language-Action Pre-Training Enables Zero-shot Cross-Embodiment Transfer](https://mlsanity.com/paper/2602.10556/) (2026)
- [Cross-Hand Latent Representation for Vision-Language-Action Models](https://mlsanity.com/paper/2603.10158/) (2026)
- [Built Different: Tactile Perception to Overcome Cross-Embodiment Capability Differences in Collaborative Manipulation](https://mlsanity.com/paper/2409.14896/) (2024)
- [Invariance Co-training for Robot Visual Generalization](https://mlsanity.com/paper/2512.05230/) (2025)
- [Morphology-Conditioned World Model for Cross-Embodiment Quadrupedal Locomotion](https://mlsanity.com/paper/2604.08780/) (2026)
- [Pick-and-place Manipulation Across Grippers Without Retraining: A Learning-optimization Diffusion Policy Approach](https://mlsanity.com/paper/2502.15613/) (2025)
- [MOTIF: Learning Action Motifs for Few-shot Cross-Embodiment Transfer](https://mlsanity.com/paper/2602.13764/) (2026)
- [Inference-stage Adaptation-projection Strategy Adapts Diffusion Policy to Cross-manipulators Scenarios](https://mlsanity.com/paper/2509.11621/) (2025)
- [Human-Humanoid Robots Cross-Embodiment Behavior-Skill Transfer Using Decomposed Adversarial Learning from Demonstration](https://mlsanity.com/paper/2412.15166/) (2024)
- [Efficient Knowledge Transfer for Jump-Starting Control Policy Learning of Multirotors through Physics-Aware Neural Architectures](https://mlsanity.com/paper/2602.15533/) (2026)
- [SCAR: Self-Supervised Continuous Action Representation Learning](https://mlsanity.com/paper/2605.16412/) (2026)
- [From Passive Video to Editable Experience: Physically Grounded Experience Synthesis for Embodied Intelligence](https://mlsanity.com/paper/2607.26903/) (2026)
- [asRoBallet: Closing the Sim2Real Gap via Friction-Aware Reinforcement Learning for Underactuated Spherical Dynamics](https://mlsanity.com/paper/2604.24916/) (2026)
- [Efficient Skill Grounding via Code Refactoring with Small Language Models](https://mlsanity.com/paper/2606.07999/) (2026)
- [One Policy, Many Embodiments: Unified Camera-Centric Action Geometry Pre-training for Heterogeneous Embodied Manipulation](https://mlsanity.com/paper/2608.26058/) (2026)
- [ZETA: A Controlled Study of Zero-Shot Cross-Embodiment VLA Transfer for Tabletop Manipulation](https://mlsanity.com/paper/2609.02546/) (2026)
- [Toward Unified Robot Learning: Bridging Representation, Vision-Language-Action, and World Models](https://mlsanity.com/paper/2609.03927/) (2026)

## Abstract (arXiv)

This paper focuses on transferring control policies between robot manipulators with different morphology. While reinforcement learning (RL) methods have shown successful results in robot manipulation tasks, transferring a trained policy from simulation to a real robot or deploying it on a robot with different states, actions, or kinematics is challenging. To achieve cross-embodiment policy transfer, our key insight is to project the state and action spaces of the source and target robots to a common latent space representation. We first introduce encoders and decoders to associate the states and actions of the source robot with a latent space. The encoders, decoders, and a latent space control policy are trained simultaneously using loss functions measuring task performance, latent dynamics consistency, and encoder-decoder ability to reconstruct the original states and actions. To transfer the learned control policy, we only need to train target encoders and decoders that align a new target domain to the latent space. We use generative adversarial training with cycle consistency and latent dynamics losses without access to the task reward or reward tuning in the target domain. We demonstrate sim-to-sim and sim-to-real manipulation policy transfer with source and target robots of different states, actions, and embodiments. The source code is available at urlhttps://github.com/ExistentialRobotics/crossembodimenttransfer.