mlsanityData to 7 Oct 2026

Robots Learn Social Skills: End-to-End Learning of Co-Speech Gesture Generation for Humanoid Robots

arXiv 1810.12541 ↗First posted 30 Oct 2018cs.ROcited by 110

Not on the map. Why?

Abstract (arXiv)

Co-speech gestures enhance interaction experiences between humans as well as between humans and robots. Existing robots use rule-based speech-gesture association, but this requires human labor and prior knowledge of experts to be implemented. We present a learning-based co-speech gesture generation that is learned from 52 h of TED talks. The proposed end-to-end neural network model consists of an encoder for speech text understanding and a decoder to generate a sequence of gestures. The model successfully produces various gestures including iconic, metaphoric, deictic, and beat gestures. In a subjective evaluation, participants reported that the gestures were human-like and matched the speech content. We also demonstrate a co-speech gesture with a NAO robot working in real time.

Key facts. Cited by 110 papers in the map (data through 7 Oct 2026). First posted on arXiv 30 Oct 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 13; 2023: 37; 2024: 25; 2025: 12; 2026: 14.

110 papers.

Sort:

By area:

  1. 2022Generating Holistic 3D Human Motion from Speechcited by 180
  2. 2022BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesiscited by 177
  3. 2023GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latentscited by 156
  4. 2023Taming Diffusion Models for Audio-Driven Co-Speech Gesture Generationcited by 128
  5. 2023One-Stage 3D Whole-Body Mesh Recovery with Component Aware Transformercited by 128
  6. 2022Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generationcited by 106
  7. 2023EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modelingcited by 101
  8. 2021Learning Speech-driven 3D Conversational Gestures from Videocited by 98
  9. 2023Human Motion Generation: A Surveycited by 95
  10. 2023DiffuseStyleGesture: Stylized Audio-Driven Co-Speech Gesture Generation with Diffusion Modelscited by 85
  11. 2023PhysHOI: Physics-Based Imitation of Dynamic Human-Object Interactioncited by 72
  12. 2023A Comprehensive Review of Data-Driven Co-Speech Gesture Generationcited by 66
38 more
  1. 2024Large Motion Model for Unified Multi-Modal Motion Generationcited by 65
  2. 2023ReMoS: 3D Motion-Conditioned Reaction Synthesis for Two-Person Interactionscited by 62
  3. 2024DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generationcited by 61
  4. 2021Speech2AffectiveGestures: Synthesizing Co-Speech Gestures with Generative Adversarial Affective Expression Learningcited by 58
  5. 2021Speech Drives Templates: Co-Speech Gesture Synthesis with Learned Templatescited by 57
  6. 2022Audio-Driven Co-Speech Gesture Video Generationcited by 54
  7. 2024MambaTalk: Efficient Holistic Gesture Synthesis with Selective State Space Modelscited by 46
  8. 2022Learning in Audio-visual Context: A Review, Analysis, and New Perspectivecited by 44
  9. 2022CLIP-Actor: Text-Driven Recommendation and Stylization for Animating Human Meshescited by 41
  10. 2024ConvoFusion: Multi-Modal Conversational Diffusion for Co-Speech Gesture Synthesiscited by 41
  11. 2022The GENEA Challenge 2022: A large evaluation of data-driven co-speech gesture generationcited by 40
  12. 2023QPGesture: Quantization-Based and Phase-Guided Motion Matching for Natural Speech-Driven Gesture Generationcited by 40
  13. 2023EmotionGesture: Audio-Driven Diverse Emotional Co-Speech 3D Gesture Generationcited by 40
  14. 2023Emotional Speech-driven 3D Body Animation via Disentangled Latent Diffusioncited by 39
  15. 2022Rhythmic Gesticulator: Rhythm-Aware Co-Speech Gesture Synthesis with Hierarchical Neural Embeddingscited by 38
  16. 2024Co-Speech Gesture Video Generation via Motion-Decoupled Diffusion Modelcited by 34
  17. 2024SemTalk: Holistic Co-speech Motion Generation with Frame-level Semantic Emphasiscited by 32
  18. 2023Evaluating gesture generation in a large-scale open challenge: The GENEA Challenge 2022cited by 29
  19. 2023The GENEA Challenge 2023: A large scale evaluation of gesture generation models in monadic and dyadic settingscited by 27
  20. 2023Diffusion-Based Co-Speech Gesture Generation Using Joint Text and Audio Representationcited by 27
  21. 2024Retrieving Semantics from the Deep: an RAG Solution for Gesture Synthesiscited by 25
  22. 2024A Comprehensive Survey on Human Video Generation: Challenges, Methods, and Insightscited by 24
  23. 2021Transflower: probabilistic autoregressive dance generation with multimodal attentioncited by 22
  24. 2023BodyFormer: Semantics-guided 3D Body Gesture Synthesis with Transformercited by 21
  25. 2024DiffTED: One-shot Audio-driven TED Talk Video Generation with Diffusion-based Co-speech Gesturescited by 19
  26. 2023Diverse 3D Hand Gesture Prediction from Body Dynamics by Bilateral Hand Disentanglementcited by 18
  27. 2023Chain of Generation: Multi-Modal Gesture Synthesis via Cascaded Conditional Controlcited by 18
  28. 2023Weakly-Supervised Emotion Transition Learning for Diverse 3D Co-speech Gesture Generationcited by 17
  29. 2021Integrated Speech and Gesture Synthesiscited by 16
  30. 2023EMoG: Synthesizing Emotive Co-speech 3D Gesture with Diffusion Modelcited by 15
  31. 2023UnifiedGesture: A Unified Gesture Synthesis Model for Multiple Skeletonscited by 15
  32. 2021Multimodal analysis of the predictability of hand-gesture propertiescited by 14
  33. 2023The DiffuseStyleGesture+ entry to the GENEA Challenge 2023cited by 14
  34. 2024Beyond Talking -- Generating Holistic 3D Human Dyadic Motion for Communicationcited by 14
  35. 2021Generating coherent spontaneous speech and gesture from textcited by 12
  36. 2021Speech2Properties2Gestures: Gesture-Property Prediction as a Tool for Generating Representational Gestures from Speechcited by 12
  37. 2022Freeform Body Motion Generation from Speechcited by 11
  38. 2022The ReprGesture entry to the GENEA Challenge 2022cited by 9

50 most cited of 110 All 110 (JSON)