mlsanityData to 14 Oct 2026

EMoG: Synthesizing Emotive Co-speech 3D Gesture with Diffusion Model

arXiv 2306.11496 ↗First posted 20 Jun 2023cs.CVcited by 15

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Although previous co-speech gesture generation methods are able to synthesize motions in line with speech content, it is still not enough to handle diverse and complicated motion distribution. The key challenges are: 1) the one-to-many nature between the speech content and gestures; 2) the correlation modeling between the body joints. In this paper, we present a novel framework (EMoG) to tackle the above challenges with denoising diffusion models: 1) To alleviate the one-to-many problem, we incorporate emotion clues to guide the generation process, making the generation much easier; 2) To model joint correlation, we propose to decompose the difficult gesture generation into two sub-problems: joint correlation modeling and temporal dynamics modeling. Then, the two sub-problems are explicitly tackled with our proposed Joint Correlation-aware transFormer (JCFormer). Through extensive evaluations, we demonstrate that our proposed method surpasses previous state-of-the-art approaches, offering substantial superiority in gesture synthesis.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 34 of its 45 references that are in the map. First posted on arXiv 20 Jun 2023.

Builds on

34 of 45 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latents26 Mar
  2. 2023Taming Diffusion Models for Audio-Driven Co-Speech Gesture Generation16 Mar

Other references · 32

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  8. 2022Scalable Diffusion Models with Transformerscited by 6,054
  9. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  10. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  11. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,333
  12. 2017Arbitrary Style Transfer in Real-time with Adaptive Instance Normalizationcited by 2,010
20 more
  1. 2022Human Motion Diffusion Modelcited by 1,080
  2. 2020Improved Techniques for Training Score-Based Generative Modelscited by 1,043
  3. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Modelcited by 780
  4. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 658
  5. 2021AI Choreographer: Music Conditioned 3D Dance Generation with AIST++cited by 485
  6. 2021FaceFormer: Speech-Driven 3D Facial Animation with Transformerscited by 209
  7. 2020Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identitycited by 202
  8. 2019Learning Individual Styles of Conversational Gesturecited by 191
  9. 2022Generating Holistic 3D Human Motion from Speechcited by 180
  10. 2022BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesiscited by 178
  11. 2018Robots Learn Social Skills: End-to-End Learning of Co-Speech Gesture Generation for Humanoid Robotscited by 110
  12. 2021Audio2Gestures: Generating Diverse Gestures from Speech Audio with Conditional Variational Autoencoderscited by 109
  13. 2022Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generationcited by 107
  14. 2021Text2Gestures: A Transformer-Based Network for Generating Emotive Body Gestures for Virtual Agentscited by 106
  15. 2021Learning Speech-driven 3D Conversational Gestures from Videocited by 98
  16. 2020Gesticulator: A framework for semantically-aware speech-driven gesture generationcited by 84
  17. 2022ZeroEGGS: Zero-shot Example-based Gesture Generation from Speechcited by 84
  18. 2021Speech2AffectiveGestures: Synthesizing Co-Speech Gestures with Generative Adversarial Affective Expression Learningcited by 58
  19. 2022Audio-Driven Co-Speech Gesture Video Generationcited by 55
  20. 2021High-Resolution Complex Scene Synthesis with Transformerscited by 31

Built on by

Cited by year

Citing papers per year: 2023: 5; 2024: 7; 2025: 3.

15 papers.

Sort:

By area:

  1. 2023State of the Art on Diffusion Models for Visual Computingcited by 127
  2. 2024Large Motion Model for Unified Multi-Modal Motion Generationcited by 65
  3. 2024MambaTalk: Efficient Holistic Gesture Synthesis with Selective State Space Modelscited by 46
  4. 2023Emotional Speech-driven 3D Body Animation via Disentangled Latent Diffusioncited by 39
  5. 2024Towards Variable and Coordinated Holistic Co-Speech Motion Generationcited by 29
  6. 2023Chain of Generation: Multi-Modal Gesture Synthesis via Cascaded Conditional Controlcited by 18
  7. 2023The DiffuseStyleGesture+ entry to the GENEA Challenge 2023cited by 14
  8. 2024Beyond Talking -- Generating Holistic 3D Human Dyadic Motion for Communicationcited by 14
  9. 2023BOTH2Hands: Inferring 3D Hands from Both Text Prompts and Body Dynamicscited by 7
  10. 2024Combo: Co-speech holistic 3D human motion generation and efficient customizable adaptation in harmonycited by 6
  11. 2024MMoFusion: Multi-modal Co-Speech Motion Generation with Diffusion Modelcited by 5
  12. 2025Text-Driven 3D Hand Motion Generation from Sign Language Datacited by 5
3 more
  1. 2025Cosh-DiT: Co-Speech Gesture Video Synthesis via Hybrid Audio-Visual Diffusion Transformerscited by 3
  2. 2024ExpGest: Expressive Speaker Generation Using Diffusion Model and Hybrid Audio-Text Guidancecited by 2
  3. 2025Evaluation of Generative Models for Emotional 3D Animation Generation in VRcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generation
  2. 2023DiffuseStyleGesture: Stylized Audio-Driven Co-Speech Gesture Generation with Diffusion Models
  3. 2024MDT-A2G: Exploring Masked Diffusion Transformers for Co-Speech Gesture Generation
  4. 2024Co-Speech Gesture Video Generation via Motion-Decoupled Diffusion Model
  5. 2023DiffMotion: Speech-Driven Gesture Synthesis Using Denoising Diffusion Model
  6. 2026Puppeteer: Object-Grounded Posture-Aware Co-Speech Gesture Generation
  7. 2023UnifiedGesture: A Unified Gesture Synthesis Model for Multiple Skeletons
  8. 2025Streaming Generation of Co-Speech Gestures via Accelerated Rolling Diffusion