mlsanityData to 7 Oct 2026

Generating Holistic 3D Human Motion from Speech

arXiv 2212.04420 ↗First posted 8 Dec 2022cs.CV cs.GRcited by 180

Not on the map. Why?

Cited by 34 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This work addresses the problem of generating 3D holistic body motions from human speech. Given a speech recording, we synthesize sequences of 3D body poses, hand gestures, and facial expressions that are realistic and diverse. To achieve this, we first build a high-quality dataset of 3D holistic body meshes with synchronous speech. We then define a novel speech-to-motion generation framework in which the face, body, and hands are modeled separately. The separated modeling stems from the fact that face articulation strongly correlates with human speech, while body poses and hand gestures are less correlated. Specifically, we employ an autoencoder for face motions, and a compositional vector-quantized variational autoencoder (VQ-VAE) for the body and hand motions. The compositional VQ-VAE is key to generating diverse results. Additionally, we propose a cross-conditional autoregressive model that generates body poses and hand gestures, leading to coherent and realistic motions. Extensive experiments and user studies demonstrate that our proposed approach achieves state-of-the-art performance both qualitatively and quantitatively. Our novel dataset and code will be released for research purposes at https://talkshow.is.tue.mpg.de.

Key facts. Cited by 180 papers in the map (data through 7 Oct 2026). Builds on 44 of its 79 references that are in the map. First posted on arXiv 8 Dec 2022.

Builds on

44 of 79 references.

References · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  3. 2016Layer Normalizationcited by 5,579
  4. 2017Neural Discrete Representation Learningcited by 4,650
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  6. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  7. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,154
  8. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
  9. 2016Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,061
  10. 2017End-to-end Recovery of Human Shape and Posecited by 883
  11. 2016Conditional Image Generation with PixelCNN Decoderscited by 859
  12. 2016Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Imagecited by 692
32 more
  1. 2019Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Setcited by 481
  2. 2020Learning an Animatable Detailed 3D Face Model from In-The-Wild Imagescited by 475
  3. 2020MakeItTalk: Speaker-Aware Talking-Head Animationcited by 310
  4. 2019Capture, Learning, and Synthesis of 3D Speaking Stylescited by 257
  5. 2021PyMAF: 3D Human Pose and Shape Regression with Pyramidal Mesh Alignment Feedback Loopcited by 250
  6. 2018Total Capture: A 3D Deformation Model for Tracking Faces, Hands, and Bodiescited by 239
  7. 2022Bailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memorycited by 210
  8. 2021FaceFormer: Speech-Driven 3D Facial Animation with Transformerscited by 208
  9. 2020Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identitycited by 199
  10. 2019Learning Individual Styles of Conversational Gesturecited by 190
  11. 2021MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglementcited by 189
  12. 2022Towards Metrical Reconstruction of Human Facescited by 165
  13. 2020Monocular Expressive Body Regression through Body-Driven Attentioncited by 155
  14. 2022PyMAF-X: Towards Well-aligned Full-body Model Regression from Monocular Imagescited by 152
  15. 2021Collaborative Regression of Expressive Bodies using Moderationcited by 137
  16. 2022Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motioncited by 125
  17. 2018Robots Learn Social Skills: End-to-End Learning of Co-Speech Gesture Generation for Humanoid Robotscited by 110
  18. 2021Audio2Gestures: Generating Diverse Gestures from Speech Audio with Conditional Variational Autoencoderscited by 109
  19. 2022Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generationcited by 106
  20. 2021Learning Speech-driven 3D Conversational Gestures from Videocited by 98
  21. 2022Multiface: A Dataset for Neural Face Renderingcited by 87
  22. 2020Gesticulator: A framework for semantically-aware speech-driven gesture generationcited by 84
  23. 2019Real-time Facial Surface Geometry from Monocular Video on Mobile GPUscited by 72
  24. 2020SMPLpix: Neural Avatars from 3D Human Modelscited by 69
  25. 2020Style Transfer for Co-Speech Gesture Animation: A Multi-Speaker Conditional-Mixture Approachcited by 64
  26. 2018VisemeNet: Audio-Driven Animator-Centric Speech Animationcited by 62
  27. 2021Speech2AffectiveGestures: Synthesizing Co-Speech Gestures with Generative Adversarial Affective Expression Learningcited by 58
  28. 2021Speech Drives Templates: Co-Speech Gesture Synthesis with Learned Templatescited by 57
  29. 2022Human-Aware Object Placement for Visual Environment Reconstructioncited by 56
  30. 2018On Learning 3D Face Morphable Model from In-the-wild Imagescited by 54
  31. 2019Analyzing Input and Output Representations for Speech-Driven Gesture Generationcited by 51
  32. 2022Freeform Body Motion Generation from Speechcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 29; 2024: 57; 2025: 56; 2026: 37.

180 papers.

Sort:

By area:

  1. 2023Effective Whole-body Pose Estimation with Two-stages Distillationcited by 307
  2. 2023Motion-X: A Large-scale 3D Expressive Whole-body Human Motion Datasetcited by 220
  3. 2022ECON: Explicit Clothed humans Optimized via Normal integrationcited by 199
  4. 2024SplattingAvatar: Realistic Real-Time Human Avatars with Mesh-Embedded Gaussian Splattingcited by 171
  5. 2023GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latentscited by 156
  6. 2023SMPLer-X: Scaling Up Expressive Human Pose and Shape Estimationcited by 154
  7. 2023TADA! Text to Animatable Digital Avatarscited by 109
  8. 2023HumanTOMATO: Text-aligned Whole-body Motion Generationcited by 103
  9. 2023EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modelingcited by 101
  10. 2023Human Motion Generation: A Surveycited by 95
  11. 2023TRACE: 5D Temporal Regression of Avatars with Dynamic Cameras in 3D Environmentscited by 89
  12. 2023DiffuseStyleGesture: Stylized Audio-Driven Co-Speech Gesture Generation with Diffusion Modelscited by 85
38 more
  1. 2024From Audio to Photoreal Embodiment: Synthesizing Humans in Conversationscited by 82
  2. 2024Move as You Say, Interact as You Can: Language-guided Human Motion Generation with Scene Affordancecited by 79
  3. 2023FaceDiffuser: Speech-Driven 3D Facial Animation Synthesis Using Diffusioncited by 77
  4. 2024CoMo: Controllable Motion Generation through Language Guided Pose Code Editingcited by 66
  5. 2024Large Motion Model for Unified Multi-Modal Motion Generationcited by 65
  6. 2024DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generationcited by 61
  7. 2024Human4DiT: 360-degree Human Video Generation with 4D Diffusion Transformercited by 61
  8. 2023Visibility Aware Human-Object Interaction Tracking from Single RGB Cameracited by 58
  9. 2024VLOGGER: Multimodal Diffusion for Embodied Avatar Synthesiscited by 48
  10. 2024MambaTalk: Efficient Holistic Gesture Synthesis with Selective State Space Modelscited by 46
  11. 2024The Language of Motion: Unifying Verbal and Non-verbal Language of 3D Human Motioncited by 46
  12. 2023NIKI: Neural Inverse Kinematics with Invertible Neural Networks for 3D Human Pose and Shape Estimationcited by 45
  13. 2023Digital Life Project: Autonomous 3D Characters with Social Intelligencecited by 43
  14. 2025GestureLSM: Latent Shortcut based Co-Speech Gesture Generation with Spatial-Temporal Modelingcited by 43
  15. 2024ConvoFusion: Multi-Modal Conversational Diffusion for Co-Speech Gesture Synthesiscited by 41
  16. 2023EmotionGesture: Audio-Driven Diverse Emotional Co-Speech 3D Gesture Generationcited by 40
  17. 2023Emotional Speech-driven 3D Body Animation via Disentangled Latent Diffusioncited by 39
  18. 2024Media2Face: Co-speech Facial Animation Generation With Multi-Modality Guidancecited by 39
  19. 2023High-Fidelity Clothed Avatar Reconstruction from a Single Imagecited by 38
  20. 2023Motion Flow Matching for Human Motion Synthesis and Editingcited by 36
  21. 2024Enabling Synergistic Full-Body Control in Prompt-Based Co-Speech Motion Generationcited by 35
  22. 2024HeadStudio: Text to Animatable Head Avatars with 3D Gaussian Splattingcited by 33
  23. 2024TANGO: Co-Speech Gesture Video Reenactment with Hierarchical Audio Motion Embedding and Diffusion Interpolationcited by 32
  24. 2024SemTalk: Holistic Co-speech Motion Generation with Frame-level Semantic Emphasiscited by 32
  25. 2025SMPLest-X: Ultimate Scaling for Expressive Human Pose and Shape Estimationcited by 31
  26. 2023SignAvatars: A Large-scale 3D Sign Language Holistic Motion Dataset and Benchmarkcited by 29
  27. 2024Towards Variable and Coordinated Holistic Co-Speech Motion Generationcited by 29
  28. 2024in2IN: Leveraging individual Information to Generate Human INteractionscited by 29
  29. 2024UniTalker: Scaling up Audio-Driven 3D Facial Animation through A Unified Modelcited by 29
  30. 2024Make-Your-Anchor: A Diffusion-based 2D Avatar Generation Frameworkcited by 26
  31. 2024Retrieving Semantics from the Deep: an RAG Solution for Gesture Synthesiscited by 25
  32. 2024MotionCraft: Crafting Whole-Body Motion with Plug-and-Play Multimodal Controlscited by 24
  33. 2024ScanTalk: 3D Talking Heads from Unregistered Scanscited by 22
  34. 20243D Face Tracking from 2D Video through Iterative Dense UV to Image Flowcited by 20
  35. 2024Signs as Tokens: A Retrieval-Enhanced Multilingual Sign Language Generatorcited by 20
  36. 2025ChainHOI: Joint-based Kinematic Chain Modeling for Human-Object Interaction Generationcited by 19
  37. 2023Chain of Generation: Multi-Modal Gesture Synthesis via Cascaded Conditional Controlcited by 18
  38. 2023XAGen: 3D Expressive Human Avatars Generationcited by 17

50 most cited of 180 All 180 (JSON)