# ImitationNet: Unsupervised Human-to-Robot Motion Retargeting via Shared Latent Space

arXiv: https://arxiv.org/abs/2309.05310 · HTML: https://mlsanity.com/paper/2309.05310/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 36 references that are in the map.
- First posted on arXiv 11 Sep 2023.

## Builds on (most cited first, 15 of 15)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Human Motion Diffusion Model](https://mlsanity.com/paper/2209.14916/) (2022)
- [Deep metric learning using Triplet network](https://mlsanity.com/paper/1412.6622/) (2014)
- [HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimation](https://mlsanity.com/paper/2011.14672/) (2020)
- [TMR: Text-to-Motion Retrieval Using Contrastive 3D Human Motion Synthesis](https://mlsanity.com/paper/2305.00976/) (2023)
- [Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Models](https://mlsanity.com/paper/2211.09707/) (2022)
- [Skeleton-Aware Networks for Deep Motion Retargeting](https://mlsanity.com/paper/2005.05732/) (2020)
- [Towards Diverse and Natural Scene-aware 3D Human Motion Synthesis](https://mlsanity.com/paper/2205.13001/) (2022)
- [Neural Kinematic Networks for Unsupervised Motion Retargetting](https://mlsanity.com/paper/1804.05653/) (2018)
- [Skinned Motion Retargeting with Residual Perception of Motion Semantics & Geometry](https://mlsanity.com/paper/2303.08658/) (2023)
- [Teleoperation of Humanoid Robots: A Survey](https://mlsanity.com/paper/2301.04317/) (2023)
- [Contact-Aware Retargeting of Skinned Motion](https://mlsanity.com/paper/2109.07431/) (2021)
- [Whole-Body Geometric Retargeting for Humanoid Robots](https://mlsanity.com/paper/1909.10080/) (2019)
- [Self-Supervised Motion Retargeting with Safety Guarantee](https://mlsanity.com/paper/2103.06447/) (2021)
- [Robust Human Motion Forecasting using Transformer-based Model](https://mlsanity.com/paper/2302.08274/) (2023)

## Built on by (most cited first, 12 of 12)

- [Learning Whole-Body Human-Humanoid Interaction from Human-Human Demonstrations](https://mlsanity.com/paper/2601.09518/) (2026)
- [I-CTRL: Imitation to Control Humanoid Robots Through Constrained Reinforcement Learning](https://mlsanity.com/paper/2405.08726/) (2024)
- [HANDI: Hand-Centric Text-and-Image Conditioned Video Generation](https://mlsanity.com/paper/2412.04189/) (2024)
- [Human Motion Unlearning](https://mlsanity.com/paper/2503.18674/) (2025)
- [Multi-Embodiment Robotic Retargeting via Guided Diffusion Model](https://mlsanity.com/paper/2505.20857/) (2025)
- [MoReFlow: Motion Retargeting Learning through Unsupervised Flow Matching](https://mlsanity.com/paper/2509.25600/) (2025)
- [BarlowWalk: Self-supervised Representation Learning for Legged Robot Terrain-adaptive Locomotion](https://mlsanity.com/paper/2508.00939/) (2025)
- [GBC: Generalized Behavior-Cloning Framework for Whole-Body Humanoid Imitation](https://mlsanity.com/paper/2508.09960/) (2025)
- [Intuitive Programming, Adaptive Task Planning, and Dynamic Role Allocation in Human-Robot Collaboration](https://mlsanity.com/paper/2511.08732/) (2025)
- [Know your limits! Optimize the robot's behavior through self-awareness](https://mlsanity.com/paper/2409.10308/) (2024)
- [Redefining Data Pairing for Motion Retargeting Leveraging a Human Body Prior](https://mlsanity.com/paper/2409.13208/) (2024)
- [A Self-Supervised Approach on Motion Calibration for Enhancing Physical Plausibility in Text-to-Motion](https://mlsanity.com/paper/2602.18199/) (2026)

## Abstract (arXiv)

This paper introduces a novel deep-learning approach for human-to-robot motion retargeting, enabling robots to mimic human poses accurately. Contrary to prior deep-learning-based works, our method does not require paired human-to-robot data, which facilitates its translation to new robots. First, we construct a shared latent space between humans and robots via adaptive contrastive learning that takes advantage of a proposed cross-domain similarity metric between the human and robot poses. Additionally, we propose a consistency term to build a common latent space that captures the similarity of the poses with precision while allowing direct robot motion control from the latent space. For instance, we can generate in-between motion through simple linear interpolation between two projected human poses. We conduct a comprehensive evaluation of robot control from diverse modalities (i.e., texts, RGB videos, and key poses), which facilitates robot control for non-expert users. Our model outperforms existing works regarding human-to-robot retargeting in terms of efficiency and precision. Finally, we implemented our method in a real robot with self-collision avoidance through a whole-body controller to showcase the effectiveness of our approach. More information on our website https://evm7.github.io/UnsH2R/