mlsanityData to 7 Oct 2026

DiffMotion: Speech-Driven Gesture Synthesis Using Denoising Diffusion Model

arXiv 2301.10047 ↗First posted 24 Jan 2023cs.GR cs.CV cs.HCcited by 22

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Speech-driven gesture synthesis is a field of growing interest in virtual human creation. However, a critical challenge is the inherent intricate one-to-many mapping between speech and gestures. Previous studies have explored and achieved significant progress with generative models. Notwithstanding, most synthetic gestures are still vastly less natural. This paper presents DiffMotion, a novel speech-driven gesture synthesis architecture based on diffusion models. The model comprises an autoregressive temporal encoder and a denoising diffusion probability Module. The encoder extracts the temporal context of the speech input and historical gestures. The diffusion module learns a parameterized Markov chain to gradually convert a simple distribution into a complex distribution and generates the gestures according to the accompanied speech. Compared with baselines, objective and subjective evaluations confirm that our approach can produce natural and diverse gesticulation and demonstrate the benefits of diffusion-based models on speech-driven gesture synthesis.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 18 of its 35 references that are in the map. First posted on arXiv 24 Jan 2023.

Builds on

18 of 35 references.

References · 18

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  3. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  4. 2016Density estimation using Real NVPcited by 2,029
  5. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  6. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,539
  7. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  8. 2014NICE: Non-linear Independent Components Estimationcited by 1,052
  9. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 924
  10. 2021SRDiff: Single Image Super-Resolution with Diffusion Probabilistic Modelscited by 415
  11. 2021Autoregressive Denoising Diffusion Models for Multivariate Probabilistic Time Series Forecastingcited by 332
  12. 2020Multivariate Probabilistic Time Series Forecasting via Conditioned Normalizing Flowscited by 142
6 more
  1. 2021Audio2Gestures: Generating Diverse Gestures from Speech Audio with Conditional Variational Autoencoderscited by 109
  2. 2021Diffusion Normalizing Flowcited by 77
  3. 2019MoGlow: Probabilistic and controllable motion synthesis using normalising flowscited by 47
  4. 2022The GENEA Challenge 2022: A large evaluation of data-driven co-speech gesture generationcited by 40
  5. 2021A large, crowdsourced evaluation of gesture generation systems on common data: The GENEA Challenge 2020cited by 38
  6. 2021A Review of Evaluation Practices of Gesture Generation in Embodied Conversational Agentscited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 8; 2025: 4; 2026: 1.

22 papers.

Sort:

By area:

  1. 2023GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latentscited by 156
  2. 2023State of the Art on Diffusion Models for Visual Computingcited by 126
  3. 2023Evaluating gesture generation in a large-scale open challenge: The GENEA Challenge 2022cited by 29
  4. 2023Diffusion-Based Co-Speech Gesture Generation Using Joint Text and Audio Representationcited by 27
  5. 2024ReinDiffuse: Crafting Physically Plausible Motions with Reinforced Diffusion Modelcited by 25
  6. 2023UnifiedGesture: A Unified Gesture Synthesis Model for Multiple Skeletonscited by 15
  7. 2023Diff-TTSG: Denoising probabilistic integrated speech and gesture synthesiscited by 13
  8. 2025Social Agent: Mastering Dyadic Nonverbal Behavior Generation via Conversational LLM Agentscited by 8
  9. 2024Speech-driven Personalized Gesture Synthetics: Harnessing Automatic Fuzzy Feature Inferencecited by 6
  10. 2024AnimateMe: 4D Facial Expressions via Diffusion Modelscited by 6
  11. 2023Towards the generation of synchronized and believable non-verbal facial behaviors of a talking virtual agentcited by 5
  12. 2024Fake it to make it: Using synthetic data to remedy the data shortage in joint multimodal speech-and-gesture synthesiscited by 5
10 more
  1. 2023Scalable Motion Style Transfer with Constrained Diffusion Generationcited by 4
  2. 2025Large Language Models for Virtual Human Gesture Selectioncited by 4
  3. 2025CoordSpeaker: Exploiting Gesture Captioning for Coordinated Caption-Empowered Co-Speech Gesture Generationcited by 4
  4. 2024Towards a GENEA Leaderboard -- an Extended, Living Benchmark for Evaluating and Advancing Conversational Motion Synthesiscited by 3
  5. 2024DiM-Gesture: Co-Speech Gesture Generation with Adaptive Layer Normalization Mamba-2 frameworkcited by 2
  6. 2024DiM-Gestor: Co-Speech Gesture Generation with Adaptive Layer Normalization Mamba-2cited by 1
  7. 2025SARGes: Semantically Aligned Reliable Gesture Generation via Intent Chaincited by 1
  8. 2023Audio is all in one: speech-driven gesture synthetics using WavLM pre-trained modelcited by 0
  9. 2024LLAniMAtion: LLAMA Driven Gesture Animationcited by 0
  10. 2026LLM-Based Selection of Incongruent Verbal and Nonverbal Behavior for Virtual Humanscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generation
  2. 2023DiffuseStyleGesture: Stylized Audio-Driven Co-Speech Gesture Generation with Diffusion Models
  3. 2023Taming Diffusion Models for Audio-Driven Co-Speech Gesture Generation
  4. 2025Cosh-DiT: Co-Speech Gesture Video Synthesis via Hybrid Audio-Visual Diffusion Transformers
  5. 2023EMoG: Synthesizing Emotive Co-speech 3D Gesture with Diffusion Model
  6. 2022Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Models
  7. 2025Streaming Generation of Co-Speech Gestures via Accelerated Rolling Diffusion
  8. 2024MDT-A2G: Exploring Masked Diffusion Transformers for Co-Speech Gesture Generation