mlsanityData to 7 Oct 2026

DiffuseStyleGesture: Stylized Audio-Driven Co-Speech Gesture Generation with Diffusion Models

arXiv 2305.04919 ↗First posted 8 May 2023cs.HC cs.MMcited by 85

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

The art of communication beyond speech there are gestures. The automatic co-speech gesture generation draws much attention in computer animation. It is a challenging task due to the diversity of gestures and the difficulty of matching the rhythm and semantics of the gesture to the corresponding speech. To address these problems, we present DiffuseStyleGesture, a diffusion model based speech-driven gesture generation approach. It generates high-quality, speech-matched, stylized, and diverse co-speech gestures based on given speeches of arbitrary length. Specifically, we introduce cross-local attention and self-attention to the gesture diffusion pipeline to generate better speech matched and realistic gestures. We then train our model with classifier-free guidance to control the gesture style by interpolation or extrapolation. Additionally, we improve the diversity of generated gestures with different initial gestures and noise. Extensive experiments show that our method outperforms recent approaches on speech-driven gesture generation. Our code, pre-trained models, and demos are available at https://github.com/YoungSeng/DiffuseStyleGesture.

Key facts. Cited by 85 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 35 of its 56 references that are in the map. First posted on arXiv 8 May 2023.

Builds on

35 of 56 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Models17 Nov

Other references · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  4. 2022Classifier-Free Diffusion Guidancecited by 5,717
  5. 2020Longformer: The Long-Document Transformercited by 3,362
  6. 2020Reformer: The Efficient Transformercited by 1,658
  7. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  8. 2022Human Motion Diffusion Modelcited by 1,062
  9. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Modelcited by 768
  10. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  11. 2022PhysDiff: Physics-Guided Human Motion Diffusion Modelcited by 365
  12. 2022MoFusion: A Framework for Denoising-Diffusion-based Motion Synthesiscited by 218
22 more
  1. 2022FLAME: Free-form Language-based Motion Synthesis & Editingcited by 217
  2. 2022Bailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memorycited by 210
  3. 2020Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identitycited by 199
  4. 2019Learning Individual Styles of Conversational Gesturecited by 190
  5. 2022Generating Holistic 3D Human Motion from Speechcited by 180
  6. 2018Robots Learn Social Skills: End-to-End Learning of Co-Speech Gesture Generation for Humanoid Robotscited by 110
  7. 2021Audio2Gestures: Generating Diverse Gestures from Speech Audio with Conditional Variational Autoencoderscited by 109
  8. 2022Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generationcited by 106
  9. 2021Learning Speech-driven 3D Conversational Gestures from Videocited by 98
  10. 2022ZeroEGGS: Zero-shot Example-based Gesture Generation from Speechcited by 83
  11. 2022VIDM: Video Implicit Diffusion Modelscited by 78
  12. 2022UDE: A Unified Driving Engine for Human Motion Generationcited by 67
  13. 2023A Comprehensive Review of Data-Driven Co-Speech Gesture Generationcited by 66
  14. 2020Style Transfer for Co-Speech Gesture Animation: A Multi-Speaker Conditional-Mixture Approachcited by 64
  15. 2022Diffusion Motion: Generate Text-Guided 3D Human Motion by Diffusion Modelcited by 41
  16. 2022The GENEA Challenge 2022: A large evaluation of data-driven co-speech gesture generationcited by 40
  17. 2021A large, crowdsourced evaluation of gesture generation systems on common data: The GENEA Challenge 2020cited by 38
  18. 2020Do Transformers Need Deep Long-Range Memorycited by 24
  19. 2023Scene-Aware 3D Multi-Human Motion Capture from a Single Cameracited by 19
  20. 2021Generating coherent spontaneous speech and gesture from textcited by 12
  21. 2022Unifying Human Motion Synthesis and Style Transfer with Denoising Diffusion Probabilistic Modelscited by 10
  22. 2022The ReprGesture entry to the GENEA Challenge 2022cited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 14; 2024: 26; 2025: 30; 2026: 15.

85 papers.

Sort:

By area:

  1. 2023State of the Art on Diffusion Models for Visual Computingcited by 126
  2. 2023EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modelingcited by 101
  3. 2023FaceDiffuser: Speech-Driven 3D Facial Animation Synthesis Using Diffusioncited by 77
  4. 2024DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generationcited by 61
  5. 2024MambaTalk: Efficient Holistic Gesture Synthesis with Selective State Space Modelscited by 46
  6. 2024The Language of Motion: Unifying Verbal and Non-verbal Language of 3D Human Motioncited by 46
  7. 2024ConvoFusion: Multi-Modal Conversational Diffusion for Co-Speech Gesture Synthesiscited by 41
  8. 2023EmotionGesture: Audio-Driven Diverse Emotional Co-Speech 3D Gesture Generationcited by 40
  9. 2023Emotional Speech-driven 3D Body Animation via Disentangled Latent Diffusioncited by 39
  10. 2024Semantic Gesticulator: Semantics-Aware Co-Speech Gesture Synthesiscited by 39
  11. 2024Enabling Synergistic Full-Body Control in Prompt-Based Co-Speech Motion Generationcited by 35
  12. 2024Co-Speech Gesture Video Generation via Motion-Decoupled Diffusion Modelcited by 34
38 more
  1. 2024TANGO: Co-Speech Gesture Video Reenactment with Hierarchical Audio Motion Embedding and Diffusion Interpolationcited by 32
  2. 2024SemTalk: Holistic Co-speech Motion Generation with Frame-level Semantic Emphasiscited by 32
  3. 2024Towards Variable and Coordinated Holistic Co-Speech Motion Generationcited by 29
  4. 2023The GENEA Challenge 2023: A large scale evaluation of gesture generation models in monadic and dyadic settingscited by 27
  5. 2023MotionScript: Natural Language Descriptions for Expressive 3D Human Motionscited by 21
  6. 2024DiffTED: One-shot Audio-driven TED Talk Video Generation with Diffusion-based Co-speech Gesturescited by 19
  7. 2023Chain of Generation: Multi-Modal Gesture Synthesis via Cascaded Conditional Controlcited by 18
  8. 2024Freetalker: Controllable Speech and Text-Driven Gesture Generation Based on Diffusion Models for Enhanced Speaker Naturalnesscited by 18
  9. 2023Weakly-Supervised Emotion Transition Learning for Diverse 3D Co-speech Gesture Generationcited by 17
  10. 2025EchoMask: Speech-Queried Attention-based Mask Modeling for Holistic Co-Speech Motion Generationcited by 17
  11. 2025SemGes: Semantics-aware Co-Speech Gesture Generation using Semantic Coherence and Relevance Learningcited by 17
  12. 2023UnifiedGesture: A Unified Gesture Synthesis Model for Multiple Skeletonscited by 15
  13. 2023The DiffuseStyleGesture+ entry to the GENEA Challenge 2023cited by 14
  14. 2023A Unified Framework for Multimodal, Multi-Part Human Motion Synthesiscited by 12
  15. 2025Mitigating Error Accumulation in Co-Speech Motion Generation via Global Rotation Diffusion and Multi-Level Constraintscited by 12
  16. 2025AudCast: Audio-Driven Human Video Generation by Cascaded Diffusion Transformerscited by 10
  17. 2025Motion-example-controlled Co-speech Gesture Generation Leveraging Large Language Modelscited by 10
  18. 2025Co3Gesture: Towards Coherent Concurrent Co-speech 3D Gesture Generation with Interactive Diffusioncited by 9
  19. 2025Social Agent: Mastering Dyadic Nonverbal Behavior Generation via Conversational LLM Agentscited by 8
  20. 2024MDT-A2G: Exploring Masked Diffusion Transformers for Co-Speech Gesture Generationcited by 7
  21. 2026PersonaGesture: Single-Reference Co-Speech Gesture Personalization for Unseen Speakerscited by 7
  22. 2024Speech-driven Personalized Gesture Synthetics: Harnessing Automatic Fuzzy Feature Inferencecited by 6
  23. 2024Combo: Co-speech holistic 3D human motion generation and efficient customizable adaptation in harmonycited by 6
  24. 2025HOP: Heterogeneous Topology-based Multimodal Entanglement for Co-Speech Gesture Generationcited by 6
  25. 2025GestureHYDRA: Semantic Co-speech Gesture Synthesis via Hybrid Modality Diffusion Transformer and Cascaded-Synchronized Retrieval-Augmented Generationcited by 6
  26. 2025ViBES: A Conversational Agent with Behaviorally-Intelligent 3D Virtual Bodycited by 6
  27. 2024MMoFusion: Multi-modal Co-Speech Motion Generation with Diffusion Modelcited by 5
  28. 2024Fake it to make it: Using synthetic data to remedy the data shortage in joint multimodal speech-and-gesture synthesiscited by 5
  29. 2023SpeechAct: Towards Generating Whole-body Motion from Speechcited by 4
  30. 2024Stereo-Talker: Audio-driven 3D Human Synthesis with Prior-Guided Mixture-of-Expertscited by 4
  31. 2025Separate to Collaborate: Dual-Stream Diffusion Model for Coordinated Piano Hand Motion Synthesiscited by 4
  32. 2025CoordSpeaker: Exploiting Gesture Captioning for Coordinated Caption-Empowered Co-Speech Gesture Generationcited by 4
  33. 2026Efficient Emotion-Aware Iconic Gesture Prediction for Robot Co-Speechcited by 4
  34. 2024Bridge to Non-Barrier Communication: Gloss-Prompted Fine-grained Cued Speech Gesture Generation with Diffusion Modelcited by 3
  35. 2024Towards a GENEA Leaderboard -- an Extended, Living Benchmark for Evaluating and Advancing Conversational Motion Synthesiscited by 3
  36. 2025Cosh-DiT: Co-Speech Gesture Video Synthesis via Hybrid Audio-Visual Diffusion Transformerscited by 3
  37. 2025DIDiffGes: Decoupled Semi-Implicit Diffusion Models for Real-time Gesture Generation from Speechcited by 3
  38. 2025OmniMotion: Multimodal Motion Generation with Continuous Masked Autoregressioncited by 3

50 most cited of 85 All 85 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Taming Diffusion Models for Audio-Driven Co-Speech Gesture Generation
  2. 2023DiffMotion: Speech-Driven Gesture Synthesis Using Denoising Diffusion Model
  3. 2023EMoG: Synthesizing Emotive Co-speech 3D Gesture with Diffusion Model
  4. 2025EasyGenNet: An Efficient Framework for Audio-Driven Gesture Video Generation Based on Diffusion Model
  5. 2023GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latents
  6. 2023LivelySpeaker: Towards Semantic-Aware Co-Speech Gesture Generation
  7. 2024CoCoGesture: Toward Coherent Co-speech 3D Gesture Generation in the Wild
  8. 2025Co-speech Gesture Video Generation via Motion-Based Graph Retrieval