# From Sign Language Generation to Humanoid Execution: Vision-Language Guided Retargeting with Collision Mitigation

arXiv: https://arxiv.org/abs/2607.17769 · HTML: https://mlsanity.com/paper/2607.17769/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 35 references that are in the map.
- First posted on arXiv 20 Jul 2026.

## Builds on (most cited first, 23 of 23)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [Improving Sign Language Translation with Monolingual Data by Sign Back-Translation](https://mlsanity.com/paper/2105.12397/) (2021)
- [AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulation](https://mlsanity.com/paper/2410.00371/) (2024)
- [Progressive Transformers for End-to-End Sign Language Production](https://mlsanity.com/paper/2004.14874/) (2020)
- [Retargeting Matters: General Motion Retargeting for Humanoid Motion Tracking](https://mlsanity.com/paper/2510.02252/) (2025)
- [Including Signed Languages in Natural Language Processing](https://mlsanity.com/paper/2105.05222/) (2021)
- [Signing at Scale: Learning to Co-Articulate Signs for Large-Scale Photo-Realistic Sign Language Production](https://mlsanity.com/paper/2203.15354/) (2022)
- [COAP: Compositional Articulated Occupancy of People](https://mlsanity.com/paper/2204.06184/) (2022)
- [Continuous 3D Multi-Channel Sign Language Production via Progressive Transformers and Mixture Density Networks](https://mlsanity.com/paper/2103.06982/) (2021)
- [Neural Sign Actors: A diffusion model for 3D sign language production from text](https://mlsanity.com/paper/2312.02702/) (2023)
- [G2P-DDM: Generating Sign Pose Sequence from Gloss Sequence with Discrete Diffusion Model](https://mlsanity.com/paper/2208.09141/) (2022)
- [VLMPC: Vision-Language Model Predictive Control for Robotic Manipulation](https://mlsanity.com/paper/2407.09829/) (2024)
- [T2S-GPT: Dynamic Vector Quantization for Autoregressive Sign Language Production from Text](https://mlsanity.com/paper/2406.07119/) (2024)
- [Harmon: Whole-Body Motion Generation of Humanoid Robots from Language Descriptions](https://mlsanity.com/paper/2410.12773/) (2024)
- [Signs as Tokens: A Retrieval-Enhanced Multilingual Sign Language Generator](https://mlsanity.com/paper/2411.17799/) (2024)
- [A Simple Baseline for Spoken Language to Sign Language Translation with 3D Avatars](https://mlsanity.com/paper/2401.04730/) (2024)
- [Sign Language Production with Latent Motion Transformer](https://mlsanity.com/paper/2312.12917/) (2023)
- [Towards AI-driven Sign Language Generation with Non-manual Markers](https://mlsanity.com/paper/2502.05661/) (2025)
- [VolumetricSMPL: A Neural Volumetric Body Model for Efficient Interactions, Contacts, and Collisions](https://mlsanity.com/paper/2506.23236/) (2025)
- [SignBot: Learning Human-to-Humanoid Sign Language Interaction](https://mlsanity.com/paper/2505.24266/) (2025)
- [CLOAF: CoLlisiOn-Aware Human Flow](https://mlsanity.com/paper/2403.09050/) (2024)
- [Select and Reorder: A Novel Approach for Neural Sign Language Production](https://mlsanity.com/paper/2404.11532/) (2024)

## Abstract (arXiv)

Recent sign language generation (SLG) systems increasingly output dense 3D body representations, which better preserve full-body kinematics and geometry for downstream embodiment on humanoid robots. However, these generated motions frequently exhibit self-intersections such as hand-hand and hand-torso penetration. While such artifacts may be tolerated in offline rendering, they become critical in humanoid execution as they lead to infeasible inverse-kinematics (IK) solutions, collisions, and unstable retargeted trajectories. We present a system-level framework that bridges SLG outputs to humanoid joint-space execution via two components. First, we introduce a volumetric SMPL-X collision-mitigation module that projects generated signing motions toward physically plausible configurations while minimally deviating from the original trajectory. Second, we propose a vision-language-guided retargeting algorithm built on an IK backbone: a VLM serves as a visual critic over rendered humanoid motion, identifies embodiment-specific failure modes, and triggers targeted task-space corrections. Our results highlight collision handling and perception-guided refinement as key missing components for reliable humanoid signing.