mlsanityData to 14 Oct 2026

SocialHumanoid: Towards Expressive Humanoid Behavior via One-Step Co-Speech Motion Generation

arXiv 2609.33311 ↗First posted 27 Sep 2026cs.RO cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Humanoid robots are increasingly expected to serve as embodied social agents that communicate naturally with humans through face-to-face interaction. During such communication, humanoid robots require body behaviors that are synchronized with speech, affectively expressive, and suitable for real-time execution. However, existing co-speech methods are primarily developed for digital humans and lack joint support for affective control and low-latency continuous generation on physical embodiments. To bridge this gap, we present SocialHumanoid, a system for expressive humanoid behavior via one-step co-speech motion generation. Given response speech and a specified affective condition, SocialHumanoid generates each full-body motion window in a single forward pass and connects successive windows through motion-history conditioning. The generated human motion is further converted online into embodiment-compatible robot references and tracked by a whole-body controller for physical execution. To provide explicit supervision for affective body expression, we further introduce AffectMoCap, a 4-hour dataset captured from two professional actors, containing synchronized speech, body motion, fine-grained hand motion, and emotion annotations. On BEAT2, SocialHumanoid achieves the best FGD among the compared generation methods, competitive speech-motion synchrony, and approximately 6× faster inference than GestureLSM under the same protocol. Perceptual evaluations further show that training with AffectMoCap improves affect recognition from generated body motion, while real-robot experiments demonstrate continuous affect-conditioned behavior and stable long-horizon execution. Our project page is https://rex0191.github.io/SocialHumanoid/.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 57 of its 66 references that are in the map. First posted on arXiv 27 Sep 2026.

Builds on

57 of 66 references.

References · 57

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,612
  3. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 1,040
  4. 2025OpenAI GPT-5 System Cardcited by 989
  5. 2022Autoregressive Image Generation using Residual Quantizationcited by 722
  6. 2023MoMask: Generative Masked Modeling of 3D Human Motionscited by 400
  7. 2023Perpetual Humanoid Control for Real-time Simulated Avatarscited by 218
  8. 2019Learning Individual Styles of Conversational Gesturecited by 191
  9. 2022Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Modelscited by 190
  10. 2022Generating Holistic 3D Human Motion from Speechcited by 180
  11. 2022BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesiscited by 178
  12. 2023GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latentscited by 158
38 more
  1. 2023Taming Diffusion Models for Audio-Driven Co-Speech Gesture Generationcited by 129
  2. 2026Qwen3-TTS Technical Reportcited by 118
  3. 2018Robots Learn Social Skills: End-to-End Learning of Co-Speech Gesture Generation for Humanoid Robotscited by 110
  4. 2025Improved Mean Flows: On the Challenges of Fastforward Generative Modelscited by 110
  5. 2021Audio2Gestures: Generating Diverse Gestures from Speech Audio with Conditional Variational Autoencoderscited by 109
  6. 2022Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generationcited by 107
  7. 2023EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modelingcited by 105
  8. 2025TWIST: Teleoperated Whole-Body Imitation Systemcited by 101
  9. 2025SONIC: Supersizing Motion Tracking for Natural Humanoid Whole-Body Controlcited by 101
  10. 2021Learning Speech-driven 3D Conversational Gestures from Videocited by 98
  11. 2024ExBody2: Advanced Expressive Humanoid Whole-Body Controlcited by 88
  12. 2022ZeroEGGS: Zero-shot Example-based Gesture Generation from Speechcited by 84
  13. 2025Retargeting Matters: General Motion Retargeting for Humanoid Motion Trackingcited by 75
  14. 2025OmniRetarget: Interaction-Preserving Data Generation for Humanoid Whole-Body Loco-Manipulation and Scene Interactioncited by 71
  15. 2024DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generationcited by 62
  16. 2024MotionGPT-2: A General-Purpose Motion-Language Model for Motion Generation and Understandingcited by 59
  17. 2024MambaTalk: Efficient Holistic Gesture Synthesis with Selective State Space Modelscited by 46
  18. 2025GestureLSM: Latent Shortcut based Co-Speech Gesture Generation with Spatial-Temporal Modelingcited by 44
  19. 2025TWIST2: Scalable, Portable, and Holistic Humanoid Data Collection Systemcited by 44
  20. 2023QPGesture: Quantization-Based and Phase-Guided Motion Matching for Natural Speech-Driven Gesture Generationcited by 40
  21. 2023Emotional Speech-driven 3D Body Animation via Disentangled Latent Diffusioncited by 39
  22. 2024Enabling Synergistic Full-Body Control in Prompt-Based Co-Speech Motion Generationcited by 35
  23. 2024SemTalk: Holistic Co-speech Motion Generation with Frame-level Semantic Emphasiscited by 32
  24. 2025SPIDER: Scalable Physics-Informed Dexterous Retargetingcited by 32
  25. 2024Towards Variable and Coordinated Holistic Co-Speech Motion Generationcited by 29
  26. 2024Retrieving Semantics from the Deep: an RAG Solution for Gesture Synthesiscited by 26
  27. 2025Embody 3D: A Large-scale Multimodal Motion and Behavior Datasetcited by 19
  28. 2025SemGes: Semantics-aware Co-Speech Gesture Generation using Semantic Coherence and Relevance Learningcited by 17
  29. 2025Mitigating Error Accumulation in Co-Speech Motion Generation via Global Rotation Diffusion and Multi-Level Constraintscited by 12
  30. 2023Co-Speech Gesture Synthesis using Discrete Gesture Token Learningcited by 9
  31. 2026MIBURI: Towards Expressive Interactive Gesture Synthesiscited by 9
  32. 2023CorrTalk: Correlation Between Hierarchical Speech and Facial Activity Variances for 3D Animationcited by 8
  33. 2025Progressive Human Motion Generation Based on Text and Few Motion Framescited by 8
  34. 2025GestureHYDRA: Semantic Co-speech Gesture Synthesis via Hybrid Modality Diffusion Transformer and Cascaded-Synchronized Retrieval-Augmented Generationcited by 7
  35. 2026PersonaGesture: Single-Reference Co-Speech Gesture Personalization for Unseen Speakerscited by 7
  36. 2026LiveGesture Streamable Co-Speech Gesture Generation Modelcited by 5
  37. 2025CoordSpeaker: Exploiting Gesture Captioning for Coordinated Caption-Empowered Co-Speech Gesture Generationcited by 4
  38. 2026CLOT: Closed-Loop Global Motion Tracking for Whole-Body Humanoid Teleoperationcited by 4

50 most cited of 57 All 57 (JSON)

Built on by

Not cited yet.