mlsanityData to 7 Oct 2026

GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latents

arXiv 2303.14613 ↗First posted 26 Mar 2023cs.CV cs.GRcited by 156

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

The automatic generation of stylized co-speech gestures has recently received increasing attention. Previous systems typically allow style control via predefined text labels or example motion clips, which are often not flexible enough to convey user intent accurately. In this work, we present GestureDiffuCLIP, a neural network framework for synthesizing realistic, stylized co-speech gestures with flexible style control. We leverage the power of the large-scale Contrastive-Language-Image-Pre-training (CLIP) model and present a novel CLIP-guided mechanism that extracts efficient style representations from multiple input modalities, such as a piece of text, an example motion clip, or a video. Our system learns a latent diffusion model to generate high-quality gestures and infuses the CLIP representations of style into the generator via an adaptive instance normalization (AdaIN) layer. We further devise a gesture-transcript alignment mechanism that ensures a semantically correct gesture generation based on contrastive learning. Our system can also be extended to allow fine-grained style control of individual body parts. We demonstrate an extensive set of examples showing the flexibility and generalizability of our model to a variety of style descriptions. In a user study, we show that our system outperforms the state-of-the-art approaches regarding human likeness, appropriateness, and style correctness.

Key facts. Cited by 156 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 46 of its 99 references that are in the map. First posted on arXiv 26 Mar 2023.

Builds on

46 of 99 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023DiffMotion: Speech-Driven Gesture Synthesis Using Denoising Diffusion Model24 Jan
  2. 2022Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Models17 Nov

Other references · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2022Classifier-Free Diffusion Guidancecited by 5,717
  9. 2017Neural Discrete Representation Learningcited by 4,650
  10. 2017Arbitrary Style Transfer in Real-time with Adaptive Instance Normalizationcited by 1,998
  11. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  12. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
32 more
  1. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,186
  2. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  3. 2022Human Motion Diffusion Modelcited by 1,062
  4. 2020Improved Techniques for Training Score-Based Generative Modelscited by 1,036
  5. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  6. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Modelcited by 768
  7. 2022Pseudo Numerical Methods for Diffusion Models on Manifoldscited by 695
  8. 2021DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulationcited by 640
  9. 2020Jukebox: A Generative Model for Musiccited by 535
  10. 2022MotionCLIP: Exposing Human Motion Generation to CLIP Spacecited by 406
  11. 2020Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identitycited by 199
  12. 2019Learning Individual Styles of Conversational Gesturecited by 190
  13. 2022Generating Holistic 3D Human Motion from Speechcited by 180
  14. 2022BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesiscited by 177
  15. 2021StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generatorscited by 164
  16. 2018Robots Learn Social Skills: End-to-End Learning of Co-Speech Gesture Generation for Humanoid Robotscited by 110
  17. 2021Audio2Gestures: Generating Diverse Gestures from Speech Audio with Conditional Variational Autoencoderscited by 109
  18. 2021Text2Gestures: A Transformer-Based Network for Generating Emotive Body Gestures for Virtual Agentscited by 106
  19. 2022Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generationcited by 106
  20. 2021Learning Speech-driven 3D Conversational Gestures from Videocited by 98
  21. 2020Unpaired Motion Style Transfer from Video to Animationcited by 94
  22. 2020Gesticulator: A framework for semantically-aware speech-driven gesture generationcited by 84
  23. 2022ZeroEGGS: Zero-shot Example-based Gesture Generation from Speechcited by 83
  24. 2023A Comprehensive Review of Data-Driven Co-Speech Gesture Generationcited by 66
  25. 2020Style Transfer for Co-Speech Gesture Animation: A Multi-Speaker Conditional-Mixture Approachcited by 64
  26. 2022Motion Puzzle: Arbitrary Motion Style Transfer by Body Partcited by 64
  27. 2021Speech2AffectiveGestures: Synthesizing Co-Speech Gestures with Generative Adversarial Affective Expression Learningcited by 58
  28. 2022Audio-Driven Co-Speech Gesture Video Generationcited by 54
  29. 2022The GENEA Challenge 2022: A large evaluation of data-driven co-speech gesture generationcited by 40
  30. 2022Rhythmic Gesticulator: Rhythm-Aware Co-Speech Gesture Synthesis with Hierarchical Neural Embeddingscited by 38
  31. 2022GANimator: Neural Motion Synthesis from a Single Sequencecited by 23
  32. 2021Speech2Properties2Gestures: Gesture-Property Prediction as a Tool for Generating Representational Gestures from Speechcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 29; 2024: 62; 2025: 38; 2026: 27.

156 papers.

Sort:

By area:

  1. 2023GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Modelscited by 255
  2. 2023State of the Art on Diffusion Models for Visual Computingcited by 126
  3. 2023Synthesizing Diverse Human Motions in 3D Indoor Scenescited by 109
  4. 2023EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modelingcited by 101
  5. 2023Inter-X: Towards Versatile Human-Human Interaction Analysiscited by 98
  6. 2023Human Motion Generation: A Surveycited by 95
  7. 2023EMDM: Efficient Motion Diffusion Model for Fast and High-Quality Motion Generationcited by 88
  8. 2024From Audio to Photoreal Embodiment: Synthesizing Humans in Conversationscited by 82
  9. 2024DartControl: A Diffusion-Based Autoregressive Motion Model for Real-Time Text-Driven Motion Controlcited by 80
  10. 2024Large Motion Model for Unified Multi-Modal Motion Generationcited by 65
  11. 2023ReMoS: 3D Motion-Conditioned Reaction Synthesis for Two-Person Interactionscited by 62
  12. 2024DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generationcited by 61
38 more
  1. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 60
  2. 2024Lodge: A Coarse to Fine Diffusion Network for Long Dance Generation Guided by the Characteristic Dance Primitivescited by 59
  3. 2023MoConVQ: Unified Physics-Based Motion Control via Scalable Discrete Representationscited by 58
  4. 2023OMG: Towards Open-vocabulary Motion Generation via Mixture of Controllerscited by 47
  5. 2024MambaTalk: Efficient Holistic Gesture Synthesis with Selective State Space Modelscited by 46
  6. 2024The Language of Motion: Unifying Verbal and Non-verbal Language of 3D Human Motioncited by 46
  7. 2024ConvoFusion: Multi-Modal Conversational Diffusion for Co-Speech Gesture Synthesiscited by 41
  8. 2023EmotionGesture: Audio-Driven Diverse Emotional Co-Speech 3D Gesture Generationcited by 40
  9. 2023Emotional Speech-driven 3D Body Animation via Disentangled Latent Diffusioncited by 39
  10. 2024Media2Face: Co-speech Facial Animation Generation With Multi-Modality Guidancecited by 39
  11. 2024Semantic Gesticulator: Semantics-Aware Co-Speech Gesture Synthesiscited by 39
  12. 2024SMooDi: Stylized Motion Diffusion Modelcited by 37
  13. 2024Enabling Synergistic Full-Body Control in Prompt-Based Co-Speech Motion Generationcited by 35
  14. 2024TANGO: Co-Speech Gesture Video Reenactment with Hierarchical Audio Motion Embedding and Diffusion Interpolationcited by 32
  15. 2024SemTalk: Holistic Co-speech Motion Generation with Frame-level Semantic Emphasiscited by 32
  16. 2024Monkey See, Monkey Do: Harnessing Self-attention in Motion Diffusion for Zero-shot Motion Transfercited by 30
  17. 2025Whole-Body Conditioned Egocentric Video Predictioncited by 30
  18. 2024Towards Variable and Coordinated Holistic Co-Speech Motion Generationcited by 29
  19. 2023Diffusion-Based Co-Speech Gesture Generation Using Joint Text and Audio Representationcited by 27
  20. 2024Retrieving Semantics from the Deep: an RAG Solution for Gesture Synthesiscited by 25
  21. 2024M2D2M: Multi-Motion Generation from Text with Discrete Diffusion Modelscited by 23
  22. 2023MotionScript: Natural Language Descriptions for Expressive 3D Human Motionscited by 21
  23. 2024Lodge++: High-quality and Long Dance Generation with Vivid Choreography Patternscited by 21
  24. 2024DiffTED: One-shot Audio-driven TED Talk Video Generation with Diffusion-based Co-speech Gesturescited by 19
  25. 2025ChainHOI: Joint-based Kinematic Chain Modeling for Human-Object Interaction Generationcited by 19
  26. 2024Freetalker: Controllable Speech and Text-Driven Gesture Generation Based on Diffusion Models for Enhanced Speaker Naturalnesscited by 18
  27. 2024Cinematographic Camera Diffusion Modelcited by 18
  28. 2023Weakly-Supervised Emotion Transition Learning for Diverse 3D Co-speech Gesture Generationcited by 17
  29. 2024Pay Attention and Move Better: Harnessing Attention for Interactive Motion Generation and Training-free Editingcited by 17
  30. 2025EchoMask: Speech-Queried Attention-based Mask Modeling for Holistic Co-Speech Motion Generationcited by 17
  31. 2025SemGes: Semantics-aware Co-Speech Gesture Generation using Semantic Coherence and Relevance Learningcited by 17
  32. 2023EMoG: Synthesizing Emotive Co-speech 3D Gesture with Diffusion Modelcited by 15
  33. 2023UnifiedGesture: A Unified Gesture Synthesis Model for Multiple Skeletonscited by 15
  34. 2024Body of Her: A Preliminary Study on End-to-End Humanoid Agentcited by 15
  35. 2025HERO: Human Reaction Generation from Videoscited by 15
  36. 2023Diff-TTSG: Denoising probabilistic integrated speech and gesture synthesiscited by 13
  37. 2023DreamTalk: When Emotional Talking Head Generation Meets Diffusion Probabilistic Modelscited by 13
  38. 2024DiffSal: Joint Audio and Video Learning for Diffusion Saliency Predictioncited by 13

50 most cited of 156 All 156 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023DiffuseStyleGesture: Stylized Audio-Driven Co-Speech Gesture Generation with Diffusion Models
  2. 2026Flexible Motion Generation from Language and Style References
  3. 2025Audio-driven Gesture Generation via Deviation Feature in the Latent Space
  4. 2023LivelySpeaker: Towards Semantic-Aware Co-Speech Gesture Generation
  5. 2025GestureLSM: Latent Shortcut based Co-Speech Gesture Generation with Spatial-Temporal Modeling
  6. 2024Co-Speech Gesture Video Generation via Motion-Decoupled Diffusion Model
  7. 20263DGesPolicy: Phoneme-Aware Holistic Co-Speech Gesture Generation Based on Action Control
  8. 2025GestureHYDRA: Semantic Co-speech Gesture Synthesis via Hybrid Modality Diffusion Transformer and Cascaded-Synchronized Retrieval-Augmented Generation