mlsanityData to 7 Oct 2026

Diffusion-Based Co-Speech Gesture Generation Using Joint Text and Audio Representation

arXiv 2309.05455 ↗First posted 11 Sep 2023eess.AS cs.HC cs.LGcited by 27

Not on the map. Why?

Abstract (arXiv)

This paper describes a system developed for the GENEA (Generation and Evaluation of Non-verbal Behaviour for Embodied Agents) Challenge 2023. Our solution builds on an existing diffusion-based motion synthesis model. We propose a contrastive speech and motion pretraining (CSMP) module, which learns a joint embedding for speech and gesture with the aim to learn a semantic coupling between these modalities. The output of the CSMP module is used as a conditioning signal in the diffusion-based gesture synthesis model in order to achieve semantically-aware co-speech gesture generation. Our entry achieved highest human-likeness and highest speech appropriateness rating among the submitted entries. This indicates that our system is a promising approach to achieve human-like co-speech gestures in agents that carry semantic meaning.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Builds on 30 of its 46 references that are in the map. First posted on arXiv 11 Sep 2023.

Builds on

30 of 46 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  7. 2022Classifier-Free Diffusion Guidancecited by 5,717
  8. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  9. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  10. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,339
  11. 2022Human Motion Diffusion Modelcited by 1,062
  12. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
18 more
  1. 2023T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representationscited by 577
  2. 2022Executing your Commands via Motion Diffusion in Latent Spacecited by 567
  3. 2023MotionGPT: Human Motion as a Foreign Languagecited by 521
  4. 2022MotionCLIP: Exposing Human Motion Generation to CLIP Spacecited by 406
  5. 2022Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Modelscited by 188
  6. 2023GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latentscited by 156
  7. 2018Robots Learn Social Skills: End-to-End Learning of Co-Speech Gesture Generation for Humanoid Robotscited by 110
  8. 2021Audio2Gestures: Generating Diverse Gestures from Speech Audio with Conditional Variational Autoencoderscited by 109
  9. 2020Gesticulator: A framework for semantically-aware speech-driven gesture generationcited by 84
  10. 2022ZeroEGGS: Zero-shot Example-based Gesture Generation from Speechcited by 83
  11. 2023A Comprehensive Review of Data-Driven Co-Speech Gesture Generationcited by 66
  12. 2019Analyzing Input and Output Representations for Speech-Driven Gesture Generationcited by 51
  13. 2020Let's Face It: Probabilistic Multi-modal Interlocutor-aware Generation of Facial Gestures in Dyadic Settingscited by 36
  14. 2023The GENEA Challenge 2023: A large scale evaluation of gesture generation models in monadic and dyadic settingscited by 27
  15. 2021The Case for Translation-Invariant Self-Attention in Transformer-Based Language Modelscited by 23
  16. 2023DiffMotion: Speech-Driven Gesture Synthesis Using Denoising Diffusion Modelcited by 22
  17. 2021HEMVIP: Human Evaluation of Multiple Videos in Parallelcited by 13
  18. 2023Diff-TTSG: Denoising probabilistic integrated speech and gesture synthesiscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 12; 2025: 6; 2026: 6.

27 papers.

Sort:

By area:

  1. 2023State of the Art on Diffusion Models for Visual Computingcited by 126
  2. 2024MambaTalk: Efficient Holistic Gesture Synthesis with Selective State Space Modelscited by 46
  3. 2025GestureLSM: Latent Shortcut based Co-Speech Gesture Generation with Spatial-Temporal Modelingcited by 43
  4. 2024Semantic Gesticulator: Semantics-Aware Co-Speech Gesture Synthesiscited by 39
  5. 2023MotionScript: Natural Language Descriptions for Expressive 3D Human Motionscited by 21
  6. 2024DiffTED: One-shot Audio-driven TED Talk Video Generation with Diffusion-based Co-speech Gesturescited by 19
  7. 2025Contextual Gesture: Co-Speech Gesture Video Generation through Context-aware Gesture Representationcited by 12
  8. 2025Intentional Gesture: Deliver Your Intentions with Gestures for Speechcited by 8
  9. 2023Unified speech and gesture synthesis using flow matchingcited by 6
  10. 2024Fake it to make it: Using synthetic data to remedy the data shortage in joint multimodal speech-and-gesture synthesiscited by 5
  11. 2026LiveGesture Streamable Co-Speech Gesture Generation Modelcited by 4
  12. 2024LM2D: Lyrics- and Music-Driven Dance Synthesiscited by 3
15 more
  1. 2024MM-Conv: A Multi-modal Conversational Dataset for Virtual Humanscited by 3
  2. 2024Towards a GENEA Leaderboard -- an Extended, Living Benchmark for Evaluating and Advancing Conversational Motion Synthesiscited by 3
  3. 2026SentiAvatar: Towards Expressive and Interactive Digital Humanscited by 3
  4. 2024Incorporating Spatial Awareness in Data-Driven Gesture Generation for Virtual Agentscited by 2
  5. 2026U-Mind: A Unified Framework for Real-Time Multimodal Interaction with Audiovisual Generationcited by 2
  6. 2025Gesture Evaluation in Virtual Realitycited by 1
  7. 2025Towards Reliable Human Evaluations in Gesture Generation: Insights from a Community-Driven State-of-the-Art Benchmarkcited by 1
  8. 2024LLAniMAtion: LLAMA Driven Gesture Animationcited by 0
  9. 2024Investigating the impact of 2D gesture representation on co-speech gesture generationcited by 0
  10. 20242D or not 2D: How Does the Dimensionality of Gesture Representation Affect 3D Co-Speech Gesture Generation?cited by 0
  11. 2024Joint Co-Speech Gesture and Expressive Talking Face Generation using Diffusion with Adapterscited by 0
  12. 2025Grounded Gesture Generation: Language, Motion, and Spacecited by 0
  13. 2026Prompt-to-Gesture: Measuring the Capabilities of Image-to-Video Deictic Gesture Generationcited by 0
  14. 2026LLM-Based Selection of Incongruent Verbal and Nonverbal Behavior for Virtual Humanscited by 0
  15. 2026InteractGesture: Progressive Chunk Guidance for Continuous Streaming Co-Speech Gesture Controlcited by 0