mlsanityData to 7 Oct 2026

Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Models

arXiv 2211.09707 ↗First posted 17 Nov 2022cs.LG cs.CV cs.GRcited by 188

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Diffusion models have experienced a surge of interest as highly expressive yet efficiently trainable probabilistic models. We show that these models are an excellent fit for synthesising human motion that co-occurs with audio, e.g., dancing and co-speech gesticulation, since motion is complex and highly ambiguous given audio, calling for a probabilistic description. Specifically, we adapt the DiffWave architecture to model 3D pose sequences, putting Conformers in place of dilated convolutions for improved modelling power. We also demonstrate control over motion style, using classifier-free guidance to adjust the strength of the stylistic expression. Experiments on gesture and dance generation confirm that the proposed method achieves top-of-the-line motion quality, with distinctive styles whose expression can be made more or less pronounced. We also synthesise path-driven locomotion using the same model architecture. Finally, we generalise the guidance procedure to obtain product-of-expert ensembles of diffusion models and demonstrate how these may be used for, e.g., style interpolation, a contribution we believe is of independent interest. See https://www.speech.kth.se/research/listen-denoise-action/ for video examples, data, and code.

Key facts. Cited by 188 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 76 of its 168 references that are in the map. First posted on arXiv 17 Nov 2022.

Builds on

76 of 168 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022BDDM: Bilateral Denoising Diffusion Models for Fast and High-Quality Speech Synthesis25 Mar
  2. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesis21 Sep

Other references · 74

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  9. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  10. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  11. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  12. 2022Classifier-Free Diffusion Guidancecited by 5,717
38 more
  1. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  2. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  3. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  4. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  5. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  6. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  7. 2022Video Diffusion Modelscited by 2,280
  8. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  9. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  10. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  11. 2021Variational Diffusion Modelscited by 1,145
  12. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  13. 2022Human Motion Diffusion Modelcited by 1,062
  14. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  15. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Modelcited by 768
  16. 2022On Distillation of Guided Diffusion Modelscited by 721
  17. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 620
  18. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 619
  19. 2022TEMOS: Generating diverse human motions from textual descriptionscited by 486
  20. 2021AI Choreographer: Music Conditioned 3D Dance Generation with AIST++cited by 479
  21. 2022TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Textscited by 357
  22. 2022MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolationcited by 307
  23. 2015A note on the evaluation of generative modelscited by 291
  24. 2017VAE with a VampPriorcited by 247
  25. 2022FLAME: Free-form Language-based Motion Synthesis & Editingcited by 217
  26. 2022Bailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memorycited by 210
  27. 2020Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identitycited by 199
  28. 2021Character Controllers Using Motion VAEscited by 182
  29. 2021Synthesis of Compositional Animations from Textual Descriptionscited by 177
  30. 2022BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesiscited by 177
  31. 2022EGSDE: Unpaired Image-to-Image Translation via Energy-Guided Stochastic Differential Equationscited by 172
  32. 2016Long Short-Term Memory-Networks for Machine Readingcited by 161
  33. 2022Diffusion Models for Video Prediction and Infillingcited by 134
  34. 2021DanceFormer: Music Conditioned 3D Dance Generation with Parametric Motion Transformercited by 111
  35. 2021Learning Speech-driven 3D Conversational Gestures from Videocited by 98
  36. 2020Learning to Generate Diverse Dance Motions with Transformercited by 85
  37. 2020Gesticulator: A framework for semantically-aware speech-driven gesture generationcited by 84
  38. 2022ZeroEGGS: Zero-shot Example-based Gesture Generation from Speechcited by 83

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 42; 2024: 54; 2025: 56; 2026: 36.

188 papers.

Sort:

By area:

  1. 2023Guided Motion Diffusion for Controllable Human Motion Synthesiscited by 221
  2. 2023GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latentscited by 156
  3. 2023Matcha-TTS: A fast TTS architecture with conditional flow matchingcited by 142
  4. 2023CG-HOI: Contact-Guided 3D Human-Object Interaction Generationcited by 137
  5. 2023State of the Art on Diffusion Models for Visual Computingcited by 126
  6. 2023Synthesizing Diverse Human Motions in 3D Indoor Scenescited by 109
  7. 2023Optimizing Diffusion Noise Can Serve As Universal Motion Priorscited by 97
  8. 2023Human Motion Generation: A Surveycited by 95
  9. 2023TLControl: Trajectory and Language Control for Human Motion Synthesiscited by 93
  10. 2023EMDM: Efficient Motion Diffusion Model for Fast and High-Quality Motion Generationcited by 88
  11. 2024Multi-Track Timeline Control for Text-Driven 3D Human Motion Generationcited by 88
  12. 2023DiffuseStyleGesture: Stylized Audio-Driven Co-Speech Gesture Generation with Diffusion Modelscited by 85
38 more
  1. 2024From Audio to Photoreal Embodiment: Synthesizing Humans in Conversationscited by 82
  2. 2023HumanMAC: Masked Motion Completion for Human Motion Predictioncited by 81
  3. 2023DiffPoseTalk: Speech-Driven Stylistic 3D Facial Animation and Head Pose Generation via Diffusion Modelscited by 81
  4. 2024DartControl: A Diffusion-Based Autoregressive Motion Model for Real-Time Text-Driven Motion Controlcited by 80
  5. 2023FaceDiffuser: Speech-Driven 3D Facial Animation Synthesis Using Diffusioncited by 77
  6. 2024Seamless Human Motion Composition with Blended Positional Encodingscited by 77
  7. 2024DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generationcited by 61
  8. 2023MoConVQ: Unified Physics-Based Motion Control via Scalable Discrete Representationscited by 58
  9. 2024Taming Diffusion Probabilistic Models for Character Controlcited by 56
  10. 2024Estimating Body and Hand Motion in an Ego-sensed Worldcited by 56
  11. 2023FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Modelscited by 55
  12. 2024THOR: Text to Human-Object Interaction Diffusion via Relation Interventioncited by 46
  13. 2025GENMO: A GENeralist Model for Human MOtioncited by 43
  14. 2023QPGesture: Quantization-Based and Phase-Guided Motion Matching for Natural Speech-Driven Gesture Generationcited by 40
  15. 2023Emotional Speech-driven 3D Body Animation via Disentangled Latent Diffusioncited by 39
  16. 2024Media2Face: Co-speech Facial Animation Generation With Multi-Modality Guidancecited by 39
  17. 2024Semantic Gesticulator: Semantics-Aware Co-Speech Gesture Synthesiscited by 39
  18. 2024SMooDi: Stylized Motion Diffusion Modelcited by 37
  19. 2023Probabilistic Human Mesh Recovery in 3D Scenes from Egocentric Viewscited by 33
  20. 2023Disentangled Clothed Avatar Generation from Text Descriptionscited by 33
  21. 2024SemTalk: Holistic Co-speech Motion Generation with Frame-level Semantic Emphasiscited by 32
  22. 2024Monkey See, Monkey Do: Harnessing Self-attention in Motion Diffusion for Zero-shot Motion Transfercited by 30
  23. 2025MotionLab: Unified Human Motion Generation and Editing via the Motion-Condition-Motion Paradigmcited by 30
  24. 2023Evaluating gesture generation in a large-scale open challenge: The GENEA Challenge 2022cited by 29
  25. 2025ARTalk: Speech-Driven 3D Head Animation via Autoregressive Modelcited by 28
  26. 2023The GENEA Challenge 2023: A large scale evaluation of gesture generation models in monadic and dyadic settingscited by 27
  27. 2023Diffusion-Based Co-Speech Gesture Generation Using Joint Text and Audio Representationcited by 27
  28. 2024HMD^2: Environment-aware Motion Generation from Single Egocentric Head-Mounted Devicecited by 26
  29. 2023NAP: Neural 3D Articulation Priorcited by 25
  30. 2024GaussianPrediction: Dynamic 3D Gaussian Prediction for Motion Extrapolation and Free View Synthesiscited by 25
  31. 2024ReinDiffuse: Crafting Physically Plausible Motions with Reinforced Diffusion Modelcited by 25
  32. 2025A Survey on Human Interaction Motion Generationcited by 25
  33. 2025UniPhys: Unified Planner and Controller with Diffusion for Flexible Physics-Based Character Controlcited by 24
  34. 2024M2D2M: Multi-Motion Generation from Text with Discrete Diffusion Modelscited by 23
  35. 2023CoMusion: Towards Consistent Stochastic Human Motion Prediction via Motion Diffusioncited by 21
  36. 2023MotionScript: Natural Language Descriptions for Expressive 3D Human Motionscited by 21
  37. 2025DuetGen: Music Driven Two-Person Dance Generation via Hierarchical Masked Modelingcited by 21
  38. 2024Programmable Motion Generation for Open-Set Motion Control Taskscited by 20

50 most cited of 188 All 188 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Unifying Human Motion Synthesis and Style Transfer with Denoising Diffusion Probabilistic Models
  2. 2023DiffMotion: Speech-Driven Gesture Synthesis Using Denoising Diffusion Model
  3. 2026From Diffusion to Flow: Efficient Motion Generation in MotionGPT3
  4. 2025StreamingTalker: Audio-driven 3D Facial Animation with Autoregressive Diffusion Model
  5. 2024Towards motion from video diffusion models
  6. 2023DiffDance: Cascaded Human Motion Diffusion Model for Dance Generation
  7. 2023Human Motion Diffusion as a Generative Prior
  8. 2022Pretrained Diffusion Models for Unified Human Motion Synthesis