mlsanityData to 7 Oct 2026

The DiffuseStyleGesture+ entry to the GENEA Challenge 2023

arXiv 2308.13879 ↗First posted 26 Aug 2023cs.HC cs.AI cs.MMcited by 14

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

In this paper, we introduce the DiffuseStyleGesture+, our solution for the Generation and Evaluation of Non-verbal Behavior for Embodied Agents (GENEA) Challenge 2023, which aims to foster the development of realistic, automated systems for generating conversational gestures. Participants are provided with a pre-processed dataset and their systems are evaluated through crowdsourced scoring. Our proposed model, DiffuseStyleGesture+, leverages a diffusion model to generate gestures automatically. It incorporates a variety of modalities, including audio, text, speaker ID, and seed gestures. These diverse modalities are mapped to a hidden space and processed by a modified diffusion model to produce the corresponding gesture for a given speech input. Upon evaluation, the DiffuseStyleGesture+ demonstrated performance on par with the top-tier models in the challenge, showing no significant differences with those models in human-likeness, appropriateness for the interlocutor, and achieving competitive performance with the best model on appropriateness for agent speech. This indicates that our model is competitive and effective in generating realistic and appropriate gestures for given speech. The code, pre-trained models, and demos are available at https://github.com/YoungSeng/DiffuseStyleGesture/tree/DiffuseStyleGesturePlus/BEAT-TWH-main.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 31 of its 43 references that are in the map. First posted on arXiv 26 Aug 2023.

Builds on

31 of 43 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023EMoG: Synthesizing Emotive Co-speech 3D Gesture with Diffusion Model20 Jun
  2. 2023DiffuseStyleGesture: Stylized Audio-Driven Co-Speech Gesture Generation with Diffusion Models8 May

Other references · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2020Reformer: The Efficient Transformercited by 1,658
  4. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  5. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  6. 2022Human Motion Diffusion Modelcited by 1,062
  7. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Modelcited by 768
  8. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  9. 2022EDGE: Editable Dance Generation From Musiccited by 284
  10. 2022MoFusion: A Framework for Denoising-Diffusion-based Motion Synthesiscited by 218
  11. 2022FLAME: Free-form Language-based Motion Synthesis & Editingcited by 217
  12. 2022Bailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memorycited by 210
17 more
  1. 2020Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identitycited by 199
  2. 2022BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesiscited by 177
  3. 2018Robots Learn Social Skills: End-to-End Learning of Co-Speech Gesture Generation for Humanoid Robotscited by 110
  4. 2021Audio2Gestures: Generating Diverse Gestures from Speech Audio with Conditional Variational Autoencoderscited by 109
  5. 2022Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generationcited by 106
  6. 2022ZeroEGGS: Zero-shot Example-based Gesture Generation from Speechcited by 83
  7. 2023A Comprehensive Review of Data-Driven Co-Speech Gesture Generationcited by 66
  8. 2019Analyzing Input and Output Representations for Speech-Driven Gesture Generationcited by 51
  9. 2022The GENEA Challenge 2022: A large evaluation of data-driven co-speech gesture generationcited by 40
  10. 2023QPGesture: Quantization-Based and Phase-Guided Motion Matching for Natural Speech-Driven Gesture Generationcited by 40
  11. 2023EmotionGesture: Audio-Driven Diverse Emotional Co-Speech 3D Gesture Generationcited by 40
  12. 2023Evaluating gesture generation in a large-scale open challenge: The GENEA Challenge 2022cited by 29
  13. 2023The GENEA Challenge 2023: A large scale evaluation of gesture generation models in monadic and dyadic settingscited by 27
  14. 2021A Review of Evaluation Practices of Gesture Generation in Embodied Conversational Agentscited by 24
  15. 2021Passing a Non-verbal Turing Test: Evaluating Gesture Animations Generated from Speechcited by 11
  16. 2023GTN-Bailando: Genre Consistent Long-Term 3D Dance Generation based on Pre-trained Genre Token Networkcited by 10
  17. 2022The ReprGesture entry to the GENEA Challenge 2022cited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 7; 2025: 4; 2026: 2.

14 papers.

Sort:

By area:

  1. 2024MambaTalk: Efficient Holistic Gesture Synthesis with Selective State Space Modelscited by 46
  2. 2023Chain of Generation: Multi-Modal Gesture Synthesis via Cascaded Conditional Controlcited by 18
  3. 2024Freetalker: Controllable Speech and Text-Driven Gesture Generation Based on Diffusion Models for Enhanced Speaker Naturalnesscited by 18
  4. 2024MDT-A2G: Exploring Masked Diffusion Transformers for Co-Speech Gesture Generationcited by 7
  5. 2026DyaDiT: A Multi-Modal Diffusion Transformer for Socially Favorable Dyadic Gesture Generationcited by 7
  6. 2024Speech-driven Personalized Gesture Synthetics: Harnessing Automatic Fuzzy Feature Inferencecited by 6
  7. 2024Fake it to make it: Using synthetic data to remedy the data shortage in joint multimodal speech-and-gesture synthesiscited by 5
  8. 2025CoordSpeaker: Exploiting Gesture Captioning for Coordinated Caption-Empowered Co-Speech Gesture Generationcited by 4
  9. 2024DiM-Gesture: Co-Speech Gesture Generation with Adaptive Layer Normalization Mamba-2 frameworkcited by 2
  10. 2024DiM-Gestor: Co-Speech Gesture Generation with Adaptive Layer Normalization Mamba-2cited by 1
  11. 2025Gesture Evaluation in Virtual Realitycited by 1
  12. 2025Towards Reliable Human Evaluations in Gesture Generation: Insights from a Community-Driven State-of-the-Art Benchmarkcited by 1
2 more
  1. 2025A conversational gesture synthesis system based on emotions and semanticscited by 0
  2. 2026SICAGE: Speaker-Independent Culture-Aware Gesture Generation using TED4C-L Datasetcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023DiffMotion: Speech-Driven Gesture Synthesis Using Denoising Diffusion Model
  2. 2023Emotional Speech-driven 3D Body Animation via Disentangled Latent Diffusion
  3. 2024DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generation
  4. 2025Inter-Diffusion Generation Model of Speakers and Listeners for Effective Communication
  5. 2025AI killed the video star. Audio-driven diffusion model for expressive talking head generation
  6. 2024Large Body Language Models
  7. 2025TalkingPose: Efficient Face and Gesture Animation with Feedback-guided Diffusion Model
  8. 2025MoDA: Multi-modal Diffusion Architecture for Talking Head Generation