mlsanityData to 14 Oct 2026

PhysDrift: Bridging the Embodiment Gap in Humanoid Co-Speech Motion Generation

arXiv 2606.19935 ↗First posted 18 Jun 2026cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

Humanoid robots require co-speech motions that are not only expressive and speech-aligned, but also physically executable under embodiment constraints. Existing co-speech generation pipelines are predominantly human-centric: motions are first generated in human-body representations such as SMPL-X and subsequently retargeted to humanoid robots. In this work, we identify a fundamental embodiment gap in this paradigm, where the mismatch between human motion manifolds and humanoid embodiment constraints disrupts embodiment consistency during motion transfer and physical execution. Through extensive analysis, we show that although retargeting can preserve coarse motion semantics, it significantly compresses motion diversity and weakens prosody-motion synchronization, limiting expressive humanoid behaviors. To address this problem, we first propose IK-EER, a prosody-preserving humanoid motion curation framework that jointly optimizes kinematic feasibility and speech-motion temporal alignment during retargeting. Building upon the curated robot-native motion dataset, we further introduce PhysDrift, an embodiment-aware co-speech motion generation framework that directly predicts executable humanoid joint trajectories from speech without relying on intermediate human-body representations. Unlike conventional human-centric pipelines, PhysDrift maintains embodiment consistency throughout both training and inference while incorporating physical regularization to stabilize robot motion dynamics. Extensive experiments and real-world humanoid deployment demonstrate that embodiment-aware robot-native generation substantially improves speech-motion alignment, physical plausibility, motion smoothness, inference efficiency, and real-time interaction capability.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 34 of its 51 references that are in the map. First posted on arXiv 18 Jun 2026.

Builds on

34 of 51 references.

References · 34

  1. 2022Flow Matching for Generative Modelingcited by 5,702
  2. 2017Neural Discrete Representation Learningcited by 4,712
  3. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,612
  4. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,214
  5. 2023Perpetual Humanoid Control for Real-time Simulated Avatarscited by 218
  6. 2020Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identitycited by 202
  7. 2022BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesiscited by 178
  8. 2018Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separationcited by 162
  9. 2024Expressive Whole-Body Control for Humanoid Robotscited by 139
  10. 2026Generative Modeling via Driftingcited by 116
  11. 2023EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modelingcited by 105
  12. 2024HOVER: Versatile Neural Whole-Body Controller for Humanoid Robotscited by 91
22 more
  1. 2022ZeroEGGS: Zero-shot Example-based Gesture Generation from Speechcited by 84
  2. 2025Retargeting Matters: General Motion Retargeting for Humanoid Motion Trackingcited by 75
  3. 2025OmniRetarget: Interaction-Preserving Data Generation for Humanoid Whole-Body Loco-Manipulation and Scene Interactioncited by 71
  4. 2024Generalizable Humanoid Manipulation with 3D Diffusion Policiescited by 52
  5. 2025KungfuBot: Physics-Based Humanoid Whole-Body Control for Learning Highly-Dynamic Skillscited by 52
  6. 2024Mobile-TeleVision: Predictive Motion Priors for Humanoid Whole-Body Controlcited by 51
  7. 2025GestureLSM: Latent Shortcut based Co-Speech Gesture Generation with Spatial-Temporal Modelingcited by 44
  8. 2023EmotionGesture: Audio-Driven Diverse Emotional Co-Speech 3D Gesture Generationcited by 40
  9. 2025UniTracker: Learning Universal Whole-Body Motion Tracker for Humanoid Robotscited by 37
  10. 2024Enabling Synergistic Full-Body Control in Prompt-Based Co-Speech Motion Generationcited by 35
  11. 2023HumanMimic: Learning Natural Locomotion and Transitions for Humanoid Robot via Wasserstein Adversarial Imitationcited by 32
  12. 2024MotionCraft: Crafting Whole-Body Motion with Plug-and-Play Multimodal Controlscited by 26
  13. 2024Harmon: Whole-Body Motion Generation of Humanoid Robots from Language Descriptionscited by 23
  14. 2025SemGes: Semantics-aware Co-Speech Gesture Generation using Semantic Coherence and Relevance Learningcited by 17
  15. 2025HOP: Heterogeneous Topology-based Multimodal Entanglement for Co-Speech Gesture Generationcited by 6
  16. 2026LiveGesture Streamable Co-Speech Gesture Generation Modelcited by 5
  17. 2025ManiDP: Manipulability-Aware Diffusion Policy for Posture-Dependent Bimanual Manipulationcited by 4
  18. 2025CoordSpeaker: Exploiting Gesture Captioning for Coordinated Caption-Empowered Co-Speech Gesture Generationcited by 4
  19. 2026ECHO: Edge-Cloud Humanoid Orchestration for Language-to-Motion Controlcited by 4
  20. 2025DIDiffGes: Decoupled Semi-Implicit Diffusion Models for Real-time Gesture Generation from Speechcited by 3
  21. 2025Hierarchical Intention-Aware Expressive Motion Generation for Humanoid Robotscited by 3
  22. 2025Realizing Text-Driven Motion Generation on NAO Robot: A Reinforcement Learning-Optimized Control Pipelinecited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.