# DiffMotion: Speech-Driven Gesture Synthesis Using Denoising Diffusion Model arXiv: https://arxiv.org/abs/2301.10047 ยท HTML: https://mlsanity.com/paper/2301.10047/ ## Key facts - Cited by 22 papers in the map (data through 14 Oct 2026). - Placed in Audio, speech and music, one of 14 directions in Diffusion models. - Builds on 18 of its 35 references that are in the map. - First posted on arXiv 24 Jan 2023. ## Placed in - Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/) ## Builds on (most cited first, 18 of 18) - [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020) - [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021) - [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015) - [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016) - [Structured Denoising Diffusion Models in Discrete State-Spaces](https://mlsanity.com/paper/2107.03006/) (2021) - [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018) - [Diffusion Models: A Comprehensive Survey of Methods and Applications](https://mlsanity.com/paper/2209.00796/) (2022) - [NICE: Non-linear Independent Components Estimation](https://mlsanity.com/paper/1410.8516/) (2014) - [Blended Diffusion for Text-driven Editing of Natural Images](https://mlsanity.com/paper/2111.14818/) (2021) - [SRDiff: Single Image Super-Resolution with Diffusion Probabilistic Models](https://mlsanity.com/paper/2104.14951/) (2021) - [Autoregressive Denoising Diffusion Models for Multivariate Probabilistic Time Series Forecasting](https://mlsanity.com/paper/2101.12072/) (2021) - [Multivariate Probabilistic Time Series Forecasting via Conditioned Normalizing Flows](https://mlsanity.com/paper/2002.06103/) (2020) - [Audio2Gestures: Generating Diverse Gestures from Speech Audio with Conditional Variational Autoencoders](https://mlsanity.com/paper/2108.06720/) (2021) - [Diffusion Normalizing Flow](https://mlsanity.com/paper/2110.07579/) (2021) - [MoGlow: Probabilistic and controllable motion synthesis using normalising flows](https://mlsanity.com/paper/1905.06598/) (2019) - [The GENEA Challenge 2022: A large evaluation of data-driven co-speech gesture generation](https://mlsanity.com/paper/2208.10441/) (2022) - [A large, crowdsourced evaluation of gesture generation systems on common data: The GENEA Challenge 2020](https://mlsanity.com/paper/2102.11617/) (2021) - [A Review of Evaluation Practices of Gesture Generation in Embodied Conversational Agents](https://mlsanity.com/paper/2101.03769/) (2021) ## Built on by (most cited first, 22 of 22) - [GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latents](https://mlsanity.com/paper/2303.14613/) (2023) - [State of the Art on Diffusion Models for Visual Computing](https://mlsanity.com/paper/2310.07204/) (2023) - [Evaluating gesture generation in a large-scale open challenge: The GENEA Challenge 2022](https://mlsanity.com/paper/2303.08737/) (2023) - [Diffusion-Based Co-Speech Gesture Generation Using Joint Text and Audio Representation](https://mlsanity.com/paper/2309.05455/) (2023) - [ReinDiffuse: Crafting Physically Plausible Motions with Reinforced Diffusion Model](https://mlsanity.com/paper/2410.07296/) (2024) - [UnifiedGesture: A Unified Gesture Synthesis Model for Multiple Skeletons](https://mlsanity.com/paper/2309.07051/) (2023) - [Diff-TTSG: Denoising probabilistic integrated speech and gesture synthesis](https://mlsanity.com/paper/2306.09417/) (2023) - [Social Agent: Mastering Dyadic Nonverbal Behavior Generation via Conversational LLM Agents](https://mlsanity.com/paper/2510.04637/) (2025) - [Speech-driven Personalized Gesture Synthetics: Harnessing Automatic Fuzzy Feature Inference](https://mlsanity.com/paper/2403.10805/) (2024) - [AnimateMe: 4D Facial Expressions via Diffusion Models](https://mlsanity.com/paper/2403.17213/) (2024) - [Towards the generation of synchronized and believable non-verbal facial behaviors of a talking virtual agent](https://mlsanity.com/paper/2311.12804/) (2023) - [Fake it to make it: Using synthetic data to remedy the data shortage in joint multimodal speech-and-gesture synthesis](https://mlsanity.com/paper/2404.19622/) (2024) - [Scalable Motion Style Transfer with Constrained Diffusion Generation](https://mlsanity.com/paper/2312.07311/) (2023) - [Large Language Models for Virtual Human Gesture Selection](https://mlsanity.com/paper/2503.14408/) (2025) - [CoordSpeaker: Exploiting Gesture Captioning for Coordinated Caption-Empowered Co-Speech Gesture Generation](https://mlsanity.com/paper/2511.22863/) (2025) - [Towards a GENEA Leaderboard -- an Extended, Living Benchmark for Evaluating and Advancing Conversational Motion Synthesis](https://mlsanity.com/paper/2410.06327/) (2024) - [DiM-Gesture: Co-Speech Gesture Generation with Adaptive Layer Normalization Mamba-2 framework](https://mlsanity.com/paper/2408.00370/) (2024) - [DiM-Gestor: Co-Speech Gesture Generation with Adaptive Layer Normalization Mamba-2](https://mlsanity.com/paper/2411.16729/) (2024) - [SARGes: Semantically Aligned Reliable Gesture Generation via Intent Chain](https://mlsanity.com/paper/2503.20202/) (2025) - [Audio is all in one: speech-driven gesture synthetics using WavLM pre-trained model](https://mlsanity.com/paper/2308.05995/) (2023) - [LLAniMAtion: LLAMA Driven Gesture Animation](https://mlsanity.com/paper/2405.08042/) (2024) - [LLM-Based Selection of Incongruent Verbal and Nonverbal Behavior for Virtual Humans](https://mlsanity.com/paper/2608.22731/) (2026) ## Abstract (arXiv) Speech-driven gesture synthesis is a field of growing interest in virtual human creation. However, a critical challenge is the inherent intricate one-to-many mapping between speech and gestures. Previous studies have explored and achieved significant progress with generative models. Notwithstanding, most synthetic gestures are still vastly less natural. This paper presents DiffMotion, a novel speech-driven gesture synthesis architecture based on diffusion models. The model comprises an autoregressive temporal encoder and a denoising diffusion probability Module. The encoder extracts the temporal context of the speech input and historical gestures. The diffusion module learns a parameterized Markov chain to gradually convert a simple distribution into a complex distribution and generates the gestures according to the accompanied speech. Compared with baselines, objective and subjective evaluations confirm that our approach can produce natural and diverse gesticulation and demonstrate the benefits of diffusion-based models on speech-driven gesture synthesis.