# Kinematic Motion Retargeting via Neural Latent Optimization for Learning Sign Language

arXiv: https://arxiv.org/abs/2103.08882 · HTML: https://mlsanity.com/paper/2103.08882/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Mar 2021.

## Built on by (most cited first, 7 of 7)

- [Natural Humanoid Robot Locomotion with Generative Motion Prior](https://mlsanity.com/paper/2503.09015/) (2025)
- [Two-Person Interaction Augmentation with Skeleton Priors](https://mlsanity.com/paper/2404.05490/) (2024)
- [Head and eye egocentric gesture recognition for human-robot interaction using eyewear cameras](https://mlsanity.com/paper/2201.11500/) (2022)
- [Signs of Language: Embodied Sign Language Fingerspelling Acquisition from Demonstrations for Human-Robot Interaction](https://mlsanity.com/paper/2209.05135/) (2022)
- [Bridging the Communication Gap: Artificial Agents Learning Sign Language through Imitation](https://mlsanity.com/paper/2406.10043/) (2024)
- [Redefining Data Pairing for Motion Retargeting Leveraging a Human Body Prior](https://mlsanity.com/paper/2409.13208/) (2024)
- [EMP: Executable Motion Prior for Humanoid Robot Standing Upper-body Motion Imitation](https://mlsanity.com/paper/2507.15649/) (2025)

## Abstract (arXiv)

Motion retargeting from a human demonstration to a robot is an effective way to reduce the professional requirements and workload of robot programming, but faces the challenges resulting from the differences between humans and robots. Traditional optimization-based methods are time-consuming and rely heavily on good initialization, while recent studies using feedforward neural networks suffer from poor generalization to unseen motions. Moreover, they neglect the topological information in human skeletons and robot structures. In this paper, we propose a novel neural latent optimization approach to address these problems. Latent optimization utilizes a decoder to establish a mapping between the latent space and the robot motion space. Afterward, the retargeting results that satisfy robot constraints can be obtained by searching for the optimal latent vector. Alongside with latent optimization, neural initialization exploits an encoder to provide a better initialization for faster and better convergence of optimization. Both the human skeleton and the robot structure are modeled as graphs to make better use of topological information. We perform experiments on retargeting Chinese sign language, which involves two arms and two hands, with additional requirements on the relative relationships among joints. Experiments include retargeting various human demonstrations to YuMi, NAO, and Pepper in the simulation environment and to YuMi in the real-world environment. Both efficiency and accuracy of the proposed method are verified.