mlsanityData to 7 Oct 2026

From Audio to Photoreal Embodiment: Synthesizing Humans in Conversations

arXiv 2401.01885 ↗First posted 3 Jan 2024cs.CVcited by 82

Not on the map. Why?

Abstract (arXiv)

We present a framework for generating full-bodied photorealistic avatars that gesture according to the conversational dynamics of a dyadic interaction. Given speech audio, we output multiple possibilities of gestural motion for an individual, including face, body, and hands. The key behind our method is in combining the benefits of sample diversity from vector quantization with the high-frequency details obtained through diffusion to generate more dynamic, expressive motion. We visualize the generated motion using highly photorealistic avatars that can express crucial nuances in gestures (e.g. sneers and smirks). To facilitate this line of research, we introduce a first-of-its-kind multi-view conversational dataset that allows for photorealistic reconstruction. Experiments show our model generates appropriate and diverse gestures, outperforming both diffusion- and VQ-only methods. Furthermore, our perceptual evaluation highlights the importance of photorealism (vs. meshes) in accurately assessing subtle motion details in conversational gestures. Code and dataset available online.

Key facts. Cited by 82 papers in the map (data through 7 Oct 2026). Builds on 33 of its 47 references that are in the map. First posted on arXiv 3 Jan 2024.

Builds on

33 of 47 references.

References · 33

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2022Classifier-Free Diffusion Guidancecited by 5,717
  3. 2017Neural Discrete Representation Learningcited by 4,650
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  6. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  7. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  8. 2022Human Motion Diffusion Modelcited by 1,062
  9. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  10. 2022EDGE: Editable Dance Generation From Musiccited by 284
  11. 2019Capture, Learning, and Synthesis of 3D Speaking Stylescited by 257
  12. 2019Neural Voice Puppetry: Audio-driven Facial Reenactmentcited by 249
21 more
  1. 2019Learning Individual Styles of Conversational Gesturecited by 190
  2. 2021MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglementcited by 189
  3. 2022Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Modelscited by 188
  4. 2021Audio-Driven Emotional Video Portraitscited by 180
  5. 2022Generating Holistic 3D Human Motion from Speechcited by 180
  6. 2022BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesiscited by 177
  7. 2018Deep Appearance Models for Face Renderingcited by 164
  8. 2023GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latentscited by 156
  9. 2019Realistic Speech-Driven Facial Animation with GANscited by 142
  10. 2022Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motioncited by 125
  11. 2021Driving-Signal Aware Full-Body Avatarscited by 81
  12. 2021Responsive Listening Head Generation: A Benchmark Dataset and Baselinecited by 67
  13. 2022Talking Head Generation with Probabilistic Audio-to-Visual Diffusion Priorscited by 55
  14. 2019Towards Social Artificial Intelligence: Nonverbal Social Signal Prediction in A Triadic Interactioncited by 48
  15. 2023Can Language Models Learn to Listen?cited by 40
  16. 2023LivelySpeaker: Towards Semantic-Aware Co-Speech Gesture Generationcited by 40
  17. 2020Body2Hands: Learning to Infer 3D Hands from Conversational Gesture Body Dynamicscited by 34
  18. 2019To React or not to React: End-to-End Visual Pose Forecasting for Personalized Avatar during Dyadic Conversationscited by 26
  19. 2018Interactive Generative Adversarial Networks for Facial Expression Generation in Dyadic Interactionscited by 15
  20. 2020APB2Face: Audio-guided face reenactment with auxiliary pose and blink signalscited by 14
  21. 2022Can gender categorization influence the perception of animated virtual humans?cited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 24; 2025: 32; 2026: 26.

82 papers.

Sort:

By area:

  1. 2024The Language of Motion: Unifying Verbal and Non-verbal Language of 3D Human Motioncited by 46
  2. 2024INFP: Audio-Driven Interactive Head Generation in Dyadic Conversationscited by 40
  3. 2024Enabling Synergistic Full-Body Control in Prompt-Based Co-Speech Motion Generationcited by 35
  4. 2024Audio-Synchronized Visual Animationcited by 32
  5. 2024GigaHands: A Massive Annotated Dataset of Bimanual Hand Activitiescited by 32
  6. 2024SemTalk: Holistic Co-speech Motion Generation with Frame-level Semantic Emphasiscited by 32
  7. 2025DualTalk: Dual-Speaker Interaction for 3D Talking Head Conversationscited by 31
  8. 2025Whole-Body Conditioned Egocentric Video Predictioncited by 30
  9. 2024Towards Open Domain Text-Driven Synthesis of Multi-Person Motionscited by 27
  10. 2024Retrieving Semantics from the Deep: an RAG Solution for Gesture Synthesiscited by 25
  11. 2024Gaussian Eigen Models for Human Headscited by 21
  12. 2024SOLAMI: Social Vision-Language-Action Modeling for Immersive Interaction with 3D Autonomous Characterscited by 18
38 more
  1. 2025SemGes: Semantics-aware Co-Speech Gesture Generation using Semantic Coherence and Relevance Learningcited by 17
  2. 2025ARIG: Autoregressive Interactive Head Generation for Real-time Conversationscited by 16
  3. 2025Embody 3D: A Large-scale Multimodal Motion and Behavior Datasetcited by 16
  4. 2025SocialGen: Modeling Multi-Human Social Interaction with Language Modelscited by 14
  5. 2024ProbTalk3D: Non-Deterministic Emotion Controllable Speech-Driven 3D Facial Animation Synthesis Using VQ-VAEcited by 13
  6. 2024Modeling and Driving Human Body Soundfields through Acoustic Primitivescited by 12
  7. 2025Cafe-Talk: Generating 3D Talking Face Animation with Multimodal Coarse- and Fine-grained Controlcited by 12
  8. 2024DEGAS: Detailed Expressions on Full-Body Gaussian Avatarscited by 10
  9. 2025Motion-example-controlled Co-speech Gesture Generation Leveraging Large Language Modelscited by 10
  10. 2025AV-Flow: Transforming Text to Audio-Visual Human-like Interactionscited by 9
  11. 2025Co3Gesture: Towards Coherent Concurrent Co-speech 3D Gesture Generation with Interactive Diffusioncited by 9
  12. 2026MIBURI: Towards Expressive Interactive Gesture Synthesiscited by 9
  13. 2024Synergy and Synchrony in Couple Dancescited by 8
  14. 2025Intentional Gesture: Deliver Your Intentions with Gestures for Speechcited by 8
  15. 2025Social Agent: Mastering Dyadic Nonverbal Behavior Generation via Conversational LLM Agentscited by 8
  16. 2024InterAct: Capture and Modelling of Realistic, Expressive and Interactive Activities between Two Persons in Daily Scenarioscited by 7
  17. 2026DyaDiT: A Multi-Modal Diffusion Transformer for Socially Favorable Dyadic Gesture Generationcited by 7
  18. 2026PersonaGesture: Single-Reference Co-Speech Gesture Personalization for Unseen Speakerscited by 7
  19. 2024Combo: Co-speech holistic 3D human motion generation and efficient customizable adaptation in harmonycited by 6
  20. 2024Llama Learns to Direct: DirectorLLM for Human-Centric Video Generationcited by 6
  21. 2025KeyVID: Keyframe-Aware Video Diffusion for Audio-Synchronized Visual Animationcited by 6
  22. 2025Audio-Driven Universal Gaussian Head Avatarscited by 6
  23. 2025ViBES: A Conversational Agent with Behaviorally-Intelligent 3D Virtual Bodycited by 6
  24. 2024Fake it to make it: Using synthetic data to remedy the data shortage in joint multimodal speech-and-gesture synthesiscited by 5
  25. 2025Audio Driven Real-Time Facial Animation for Social Telepresencecited by 5
  26. 2025DyStream: Streaming Dyadic Talking Heads Generation via Flow Matching-based Autoregressive Modelcited by 5
  27. 2024TalkinNeRF: Animatable Neural Fields for Full-Body Talking Humanscited by 4
  28. 2024Human Action CLIPs: Detecting AI-generated Human Motioncited by 4
  29. 2025Exploring Timeline Control for Facial Motion Generationcited by 4
  30. 2024Towards a GENEA Leaderboard -- an Extended, Living Benchmark for Evaluating and Advancing Conversational Motion Synthesiscited by 3
  31. 2025LSF-Animation: Label-Free Speech-Driven Facial Animation via Implicit Feature Representationcited by 3
  32. 2026PolySLGen: Online Multimodal Speaking-Listening Reaction Generation in Polyadic Interactioncited by 3
  33. 2024Massively Multi-Person 3D Human Motion Forecasting with Scene Contextcited by 2
  34. 2025Towards Privacy-aware Mental Health AI Models: Advances, Challenges, and Opportunitiescited by 2
  35. 2025ExGes: Expressive Human Motion Retrieval and Modulation for Audio-Driven Gesture Synthesiscited by 2
  36. 2025Scaling Up Audio-Synchronized Visual Animation: An Efficient Training Paradigmcited by 2
  37. 2025InterAct: A Large-Scale Dataset of Dynamic, Expressive and Interactive Activities between Two People in Daily Scenarioscited by 2
  38. 2026A Platform for Interactive AI Character Experiencescited by 2

50 most cited of 82 All 82 (JSON)