mlsanityData to 7 Oct 2026

Gesticulator: A framework for semantically-aware speech-driven gesture generation

arXiv 2001.09326 ↗First posted 25 Jan 2020cs.HC cs.LG eess.AScited by 84

Not on the map. Why?

Abstract (arXiv)

During speech, people spontaneously gesticulate, which plays a key role in conveying information. Similarly, realistic co-speech gestures are crucial to enable natural and smooth interactions with social agents. Current end-to-end co-speech gesture generation systems use a single modality for representing speech: either audio or text. These systems are therefore confined to producing either acoustically-linked beat gestures or semantically-linked gesticulation (e.g., raising a hand when saying "high"): they cannot appropriately learn to generate both gesture types. We present a model designed to produce arbitrary beat and semantic gestures together. Our deep-learning based model takes both acoustic and semantic representations of speech as input, and generates gestures as a sequence of joint angle rotations as output. The resulting gestures can be applied to both virtual agents and humanoid robots. Subjective and objective evaluations confirm the success of our approach. The code and video are available at the project page https://svito-zar.github.io/gesticulator .

Key facts. Cited by 84 papers in the map (data through 7 Oct 2026). First posted on arXiv 25 Jan 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 14; 2023: 23; 2024: 17; 2025: 11; 2026: 11.

84 papers.

Sort:

By area:

  1. 2022Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Modelscited by 188
  2. 2022Generating Holistic 3D Human Motion from Speechcited by 180
  3. 2022BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesiscited by 177
  4. 2023GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latentscited by 156
  5. 2022Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generationcited by 106
  6. 2021Learning Speech-driven 3D Conversational Gestures from Videocited by 98
  7. 2023Human Motion Generation: A Surveycited by 95
  8. 2022ZeroEGGS: Zero-shot Example-based Gesture Generation from Speechcited by 83
  9. 2023A Comprehensive Review of Data-Driven Co-Speech Gesture Generationcited by 66
  10. 2024DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generationcited by 61
  11. 2021Speech2AffectiveGestures: Synthesizing Co-Speech Gestures with Generative Adversarial Affective Expression Learningcited by 58
  12. 2022Audio-Driven Co-Speech Gesture Video Generationcited by 54
38 more
  1. 2022Learning in Audio-visual Context: A Review, Analysis, and New Perspectivecited by 44
  2. 2024ConvoFusion: Multi-Modal Conversational Diffusion for Co-Speech Gesture Synthesiscited by 41
  3. 2022The GENEA Challenge 2022: A large evaluation of data-driven co-speech gesture generationcited by 40
  4. 2023QPGesture: Quantization-Based and Phase-Guided Motion Matching for Natural Speech-Driven Gesture Generationcited by 40
  5. 2023Can Language Models Learn to Listen?cited by 40
  6. 2024Semantic Gesticulator: Semantics-Aware Co-Speech Gesture Synthesiscited by 39
  7. 2022Rhythmic Gesticulator: Rhythm-Aware Co-Speech Gesture Synthesis with Hierarchical Neural Embeddingscited by 38
  8. 2024SemTalk: Holistic Co-speech Motion Generation with Frame-level Semantic Emphasiscited by 32
  9. 2023Evaluating gesture generation in a large-scale open challenge: The GENEA Challenge 2022cited by 29
  10. 2023The GENEA Challenge 2023: A large scale evaluation of gesture generation models in monadic and dyadic settingscited by 27
  11. 2023Diffusion-Based Co-Speech Gesture Generation Using Joint Text and Audio Representationcited by 27
  12. 2023BodyFormer: Semantics-guided 3D Body Gesture Synthesis with Transformercited by 21
  13. 2024DiffTED: One-shot Audio-driven TED Talk Video Generation with Diffusion-based Co-speech Gesturescited by 19
  14. 2022Audio-driven Neural Gesture Reenactment with Video Motion Graphscited by 18
  15. 2025SemGes: Semantics-aware Co-Speech Gesture Generation using Semantic Coherence and Relevance Learningcited by 17
  16. 2021Integrated Speech and Gesture Synthesiscited by 16
  17. 2023Multiple Appropriate Facial Reaction Generation in Dyadic Interaction Settings: What, Why and How?cited by 16
  18. 2021Joint Audio-Text Model for Expressive Speech-Driven 3D Facial Animationcited by 15
  19. 2023EMoG: Synthesizing Emotive Co-speech 3D Gesture with Diffusion Modelcited by 15
  20. 2023UnifiedGesture: A Unified Gesture Synthesis Model for Multiple Skeletonscited by 15
  21. 2021Multimodal analysis of the predictability of hand-gesture propertiescited by 14
  22. 2024Beyond Talking -- Generating Holistic 3D Human Dyadic Motion for Communicationcited by 14
  23. 2023Diff-TTSG: Denoising probabilistic integrated speech and gesture synthesiscited by 13
  24. 2021Graph-based Normalizing Flow for Human Motion Generation and Reconstructioncited by 12
  25. 2021Speech2Properties2Gestures: Gesture-Property Prediction as a Tool for Generating Representational Gestures from Speechcited by 12
  26. 2023AgentAvatar: Disentangling Planning, Driving and Rendering for Photorealistic Avatar Agentscited by 11
  27. 2025Motion-example-controlled Co-speech Gesture Generation Leveraging Large Language Modelscited by 10
  28. 2022The ReprGesture entry to the GENEA Challenge 2022cited by 9
  29. 2023Co-Speech Gesture Synthesis using Discrete Gesture Token Learningcited by 9
  30. 2026MIBURI: Towards Expressive Interactive Gesture Synthesiscited by 9
  31. 2025Social Agent: Mastering Dyadic Nonverbal Behavior Generation via Conversational LLM Agentscited by 8
  32. 2022Zero-Shot Style Transfer for Gesture Animation driven by Text and Speech using Adversarial Disentanglement of Multimodal Style Encodingcited by 7
  33. 2026PersonaGesture: Single-Reference Co-Speech Gesture Personalization for Unseen Speakerscited by 7
  34. 2023Large language models in textual analysis for gesture selectioncited by 6
  35. 2024Speech-driven Personalized Gesture Synthetics: Harnessing Automatic Fuzzy Feature Inferencecited by 6
  36. 2025GestureHYDRA: Semantic Co-speech Gesture Synthesis via Hybrid Modality Diffusion Transformer and Cascaded-Synchronized Retrieval-Augmented Generationcited by 6
  37. 2023Towards the generation of synchronized and believable non-verbal facial behaviors of a talking virtual agentcited by 5
  38. 2024Fake it to make it: Using synthetic data to remedy the data shortage in joint multimodal speech-and-gesture synthesiscited by 5

50 most cited of 84 All 84 (JSON)