mlsanityData to 7 Oct 2026

DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generation

arXiv 2401.04747 ↗First posted 9 Jan 2024cs.SD cs.AI cs.CVcited by 61

Not on the map. Why?

Abstract (arXiv)

We propose DiffSHEG, a Diffusion-based approach for Speech-driven Holistic 3D Expression and Gesture generation with arbitrary length. While previous works focused on co-speech gesture or expression generation individually, the joint generation of synchronized expressions and gestures remains barely explored. To address this, our diffusion-based co-speech motion generation transformer enables uni-directional information flow from expression to gesture, facilitating improved matching of joint expression-gesture distributions. Furthermore, we introduce an outpainting-based sampling strategy for arbitrary long sequence generation in diffusion models, offering flexibility and computational efficiency. Our method provides a practical solution that produces high-quality synchronized expression and gesture generation driven by speech. Evaluated on two public datasets, our approach achieves state-of-the-art performance both quantitatively and qualitatively. Additionally, a user study confirms the superiority of DiffSHEG over prior approaches. By enabling the real-time generation of expressive and synchronized motions, DiffSHEG showcases its potential for various applications in the development of digital humans and embodied agents.

Key facts. Cited by 61 papers in the map (data through 7 Oct 2026). Builds on 33 of its 56 references that are in the map. First posted on arXiv 9 Jan 2024.

Builds on

33 of 56 references.

References · 33

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2020Denoising Diffusion Implicit Modelscited by 9,655
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  5. 2017Arbitrary Style Transfer in Real-time with Adaptive Instance Normalizationcited by 1,998
  6. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
  7. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,419
  8. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Modelcited by 768
  9. 2021AI Choreographer: Music Conditioned 3D Dance Generation with AIST++cited by 479
  10. 2022SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animationcited by 380
  11. 2018Efficient Attention: Attention with Linear Complexitiescited by 366
  12. 2019Capture, Learning, and Synthesis of 3D Speaking Stylescited by 257
21 more
  1. 2021FaceFormer: Speech-Driven 3D Facial Animation with Transformerscited by 208
  2. 2020Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identitycited by 199
  3. 2021MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglementcited by 189
  4. 2022Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Modelscited by 188
  5. 2023CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Priorcited by 183
  6. 2022Generating Holistic 3D Human Motion from Speechcited by 180
  7. 2022BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesiscited by 177
  8. 2023GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latentscited by 156
  9. 2023Taming Diffusion Models for Audio-Driven Co-Speech Gesture Generationcited by 128
  10. 2018Robots Learn Social Skills: End-to-End Learning of Co-Speech Gesture Generation for Humanoid Robotscited by 110
  11. 2021Text2Gestures: A Transformer-Based Network for Generating Emotive Body Gestures for Virtual Agentscited by 106
  12. 2022Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generationcited by 106
  13. 2021Learning Speech-driven 3D Conversational Gestures from Videocited by 98
  14. 2023DiffuseStyleGesture: Stylized Audio-Driven Co-Speech Gesture Generation with Diffusion Modelscited by 85
  15. 2020Gesticulator: A framework for semantically-aware speech-driven gesture generationcited by 84
  16. 2021Speech2AffectiveGestures: Synthesizing Co-Speech Gestures with Generative Adversarial Affective Expression Learningcited by 58
  17. 2017Vid2speech: Speech Reconstruction from Silent Videocited by 53
  18. 2022Rhythmic Gesticulator: Rhythm-Aware Co-Speech Gesture Synthesis with Hierarchical Neural Embeddingscited by 38
  19. 2020Modality Dropout for Improved Performance-driven Talking Facescited by 32
  20. 2023MODA: Mapping-Once Audio-driven Portrait Animation with Dual Attentionscited by 29
  21. 2022Real-time Streaming Video Denoising with Bidirectional Bufferscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 27; 2026: 21.

61 papers.

Sort:

By area:

  1. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 914
  2. 2024MambaTalk: Efficient Holistic Gesture Synthesis with Selective State Space Modelscited by 46
  3. 2025GestureLSM: Latent Shortcut based Co-Speech Gesture Generation with Spatial-Temporal Modelingcited by 43
  4. 2024SemTalk: Holistic Co-speech Motion Generation with Frame-level Semantic Emphasiscited by 32
  5. 2024MotionCraft: Crafting Whole-Body Motion with Plug-and-Play Multimodal Controlscited by 24
  6. 2025EchoMask: Speech-Queried Attention-based Mask Modeling for Holistic Co-Speech Motion Generationcited by 17
  7. 2025SemGes: Semantics-aware Co-Speech Gesture Generation using Semantic Coherence and Relevance Learningcited by 17
  8. 2024ProbTalk3D: Non-Deterministic Emotion Controllable Speech-Driven 3D Facial Animation Synthesis Using VQ-VAEcited by 13
  9. 2025Contextual Gesture: Co-Speech Gesture Video Generation through Context-aware Gesture Representationcited by 12
  10. 2025Mitigating Error Accumulation in Co-Speech Motion Generation via Global Rotation Diffusion and Multi-Level Constraintscited by 12
  11. 2025AudCast: Audio-Driven Human Video Generation by Cascaded Diffusion Transformerscited by 10
  12. 2025Co3Gesture: Towards Coherent Concurrent Co-speech 3D Gesture Generation with Interactive Diffusioncited by 9
38 more
  1. 2025Intentional Gesture: Deliver Your Intentions with Gestures for Speechcited by 8
  2. 2025Social Agent: Mastering Dyadic Nonverbal Behavior Generation via Conversational LLM Agentscited by 8
  3. 2024RapVerse: Coherent Vocals and Whole-Body Motions Generations from Textcited by 7
  4. 2024MDT-A2G: Exploring Masked Diffusion Transformers for Co-Speech Gesture Generationcited by 7
  5. 2026PersonaGesture: Single-Reference Co-Speech Gesture Personalization for Unseen Speakerscited by 7
  6. 2024Combo: Co-speech holistic 3D human motion generation and efficient customizable adaptation in harmonycited by 6
  7. 2025GestureHYDRA: Semantic Co-speech Gesture Synthesis via Hybrid Modality Diffusion Transformer and Cascaded-Synchronized Retrieval-Augmented Generationcited by 6
  8. 2025Towards High-fidelity 3D Talking Avatar with Personalized Dynamic Texturecited by 5
  9. 2025TalkCuts: A Large-Scale Dataset for Multi-Shot Human Speech Video Generationcited by 4
  10. 2025CoordSpeaker: Exploiting Gesture Captioning for Coordinated Caption-Empowered Co-Speech Gesture Generationcited by 4
  11. 2026SemConFlow: Semantic Grounding of Holistic Co-Speech Gesture Generation with Contrastive Flow-Matchingcited by 4
  12. 2026LiveGesture Streamable Co-Speech Gesture Generation Modelcited by 4
  13. 2024Towards a GENEA Leaderboard -- an Extended, Living Benchmark for Evaluating and Advancing Conversational Motion Synthesiscited by 3
  14. 2025OmniMotion: Multimodal Motion Generation with Continuous Masked Autoregressioncited by 3
  15. 2025fMRI2GES: Co-speech Gesture Reconstruction from fMRI Signal with Dual Brain Decoding Alignmentcited by 3
  16. 2026SentiAvatar: Towards Expressive and Interactive Digital Humanscited by 3
  17. 2026RoboGesture: Real-Time Semantic-aligned Co-Speech Gestures Generation for Humanoid Interactioncited by 3
  18. 2024CoCoGesture: Toward Coherent Co-speech 3D Gesture Generation in the Wildcited by 2
  19. 2024Artificial Intelligence for Biomedical Video Generationcited by 2
  20. 2025ExGes: Expressive Human Motion Retrieval and Modulation for Audio-Driven Gesture Synthesiscited by 2
  21. 2025OmniMotion-X: Versatile Multimodal Whole-Body Motion Generationcited by 2
  22. 2026HandX: Scaling Bimanual Motion and Interaction Generationcited by 2
  23. 2026BiTDiff: Fine-Grained 3D Conducting Motion Generation via BiMamba-Transformer Diffusioncited by 2
  24. 2024LES-Talker: Fine-Grained Emotion Editing for Talking Head Generation in Linear Emotion Spacecited by 1
  25. 2025Streaming Generation of Co-Speech Gestures via Accelerated Rolling Diffusioncited by 1
  26. 2025MAG: Multi-Modal Aligned Autoregressive Co-Speech Gesture Generation without Vector Quantizationcited by 1
  27. 2025SARGes: Semantically Aligned Reliable Gesture Generation via Intent Chaincited by 1
  28. 2025MMGT: Motion Mask Guided Two-Stage Network for Co-Speech Gesture Video Generationcited by 1
  29. 2025Input-Aware Sparse Attention for Real-Time Co-Speech Video Generationcited by 1
  30. 2025Towards Reliable Human Evaluations in Gesture Generation: Insights from a Community-Driven State-of-the-Art Benchmarkcited by 1
  31. 2026SmoothSync: Dual-Stream Diffusion Transformers for Jitter-Robust Beat-Synchronized Gesture Generation from Quantized Audiocited by 1
  32. 20263DGesPolicy: Phoneme-Aware Holistic Co-Speech Gesture Generation Based on Action Controlcited by 1
  33. 2026PersonaGest: Personalized Co-Speech Gesture Generation with Semantic-Guided Hierarchical Motion Representationcited by 1
  34. 2026UMo: Unified Sparse Motion Modeling for Real-Time Co-Speech Avatarscited by 1
  35. 2026EchoAvatar: Real-time Generative Avatar Animation from Audio Streamscited by 1
  36. 2026Puppeteer: Object-Grounded Posture-Aware Co-Speech Gesture Generationcited by 1
  37. 2024Joint Co-Speech Gesture and Expressive Talking Face Generation using Diffusion with Adapterscited by 0
  38. 2025ReCoM: Realistic Co-Speech Motion Generation with Recurrent Embedded Transformercited by 0

50 most cited of 61 All 61 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023DiffMotion: Speech-Driven Gesture Synthesis Using Denoising Diffusion Model
  2. 2023EMoG: Synthesizing Emotive Co-speech 3D Gesture with Diffusion Model
  3. 2025Cosh-DiT: Co-Speech Gesture Video Synthesis via Hybrid Audio-Visual Diffusion Transformers
  4. 2024DiffTED: One-shot Audio-driven TED Talk Video Generation with Diffusion-based Co-speech Gestures
  5. 2023C2G2: Controllable Co-speech Gesture Generation with Latent Diffusion Model
  6. 2024Co-Speech Gesture Video Generation via Motion-Decoupled Diffusion Model
  7. 2025EasyGenNet: An Efficient Framework for Audio-Driven Gesture Video Generation Based on Diffusion Model
  8. 2025Inter-Diffusion Generation Model of Speakers and Listeners for Effective Communication