mlsanityData to 7 Oct 2026

Capture, Learning, and Synthesis of 3D Speaking Styles

arXiv 1905.03079 ↗First posted 8 May 2019cs.CVcited by 257

Not on the map. Why?

Cited by 22 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Audio-driven 3D facial animation has been widely explored, but achieving realistic, human-like performance is still unsolved. This is due to the lack of available 3D datasets, models, and standard evaluation metrics. To address this, we introduce a unique 4D face dataset with about 29 minutes of 4D scans captured at 60 fps and synchronized audio from 12 speakers. We then train a neural network on our dataset that factors identity from facial motion. The learned model, VOCA (Voice Operated Character Animation) takes any speech signal as input - even speech in languages other than English - and realistically animates a wide range of adult faces. Conditioning on subject labels during training allows the model to learn a variety of realistic speaking styles. VOCA also provides animator controls to alter speaking style, identity-dependent facial shape, and pose (i.e. head, jaw, and eyeball rotations) during animation. To our knowledge, VOCA is the only realistic 3D facial animation model that is readily applicable to unseen subjects without retargeting. This makes VOCA suitable for tasks like in-game video, virtual reality avatars, or any scenario in which the speaker, speech, or language is not known in advance. We make the dataset and model available for research purposes at http://voca.is.tue.mpg.de.

Key facts. Cited by 257 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 23; 2022: 25; 2023: 69; 2024: 53; 2025: 44; 2026: 43.

257 papers.

Sort:

By area:

  1. 2022TEMOS: Generating diverse human motions from textual descriptionscited by 486
  2. 2021AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesiscited by 339
  3. 2021I M Avatar: Implicit Morphable Head Avatars from Videoscited by 230
  4. 2024A Survey on 3D Gaussian Splattingcited by 221
  5. 2021FaceFormer: Speech-Driven 3D Facial Animation with Transformerscited by 208
  6. 2023NeRSemble: Multi-view Radiance Field Reconstruction of Human Headscited by 190
  7. 2021MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglementcited by 189
  8. 2023CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Priorcited by 183
  9. 2022Generating Holistic 3D Human Motion from Speechcited by 180
  10. 2022BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesiscited by 177
  11. 2022StyleHEAT: One-Shot High-Resolution Editable Talking Face Generation via Pre-trained StyleGANcited by 169
  12. 2023EmoTalk: Speech-Driven Emotional Disentanglement for 3D Face Animationcited by 141
38 more
  1. 2023DiffTalk: Crafting Diffusion Models for Generalized Audio-Driven Portraits Animationcited by 136
  2. 2023GeneFace: Generalized and High-Fidelity Audio-Driven 3D Talking Face Synthesiscited by 131
  3. 2022Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generationcited by 119
  4. 2023EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modelingcited by 101
  5. 2021Learning Speech-driven 3D Conversational Gestures from Videocited by 98
  6. 2023Emotional Speech-Driven Animation with Content-Emotion Disentanglementcited by 98
  7. 2022Learning Dynamic Facial Radiance Fields for Few-Shot Talking Head Synthesiscited by 97
  8. 2021One-shot Talking Face Generation from Single-speaker Audio-Visual Correlation Learningcited by 87
  9. 2023A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need?cited by 84
  10. 2024From Audio to Photoreal Embodiment: Synthesizing Humans in Conversationscited by 82
  11. 2022Real-time Neural Radiance Talking Portrait Synthesis via Audio-spatial Decompositioncited by 81
  12. 2023DiffPoseTalk: Speech-Driven Stylistic 3D Facial Animation and Head Pose Generation via Diffusion Modelscited by 81
  13. 2022DFA-NeRF: Personalized Talking Head Generation via Disentangled Face Attributes Neural Renderingcited by 80
  14. 2023FaceDiffuser: Speech-Driven 3D Facial Animation Synthesis Using Diffusioncited by 77
  15. 2022Imitator: Personalized Speech-driven 3D Facial Animationcited by 73
  16. 2021LipSync3D: Data-Efficient Learning of Personalized 3D Talking Faces from Video using Pose and Lighting Normalizationcited by 67
  17. 2023SelfTalk: A Self-Supervised Commutative Training Diagram to Comprehend 3D Talking Facescited by 67
  18. 2024DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generationcited by 61
  19. 2021Imitating Arbitrary Talking Style for Realistic Audio-DrivenTalking Face Synthesiscited by 56
  20. 2023FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Modelscited by 55
  21. 2022Audio-Driven Co-Speech Gesture Video Generationcited by 54
  22. 2024VLOGGER: Multimodal Diffusion for Embodied Avatar Synthesiscited by 48
  23. 2021Write-a-speaker: Text-based Emotional and Rhythmic Talking-head Generationcited by 46
  24. 2023RenderMe-360: A Large Digital Asset Library and Benchmarks Towards High-fidelity Head Avatarscited by 45
  25. 2024Media2Face: Co-speech Facial Animation Generation With Multi-Modality Guidancecited by 39
  26. 2021Live Speech Portraits: Real-Time Photorealistic Talking-Head Animationcited by 36
  27. 2021FaceScape: 3D Facial Dataset and Benchmark for Single-View 3D Face Reconstructioncited by 34
  28. 2024Dyadic Interaction Modeling for Social Behavior Generationcited by 32
  29. 2023MedShapeNet -- A Large-Scale Dataset of 3D Medical Shapes for Computer Visioncited by 31
  30. 2025DualTalk: Dual-Speaker Interaction for 3D Talking Head Conversationscited by 31
  31. 2022DialogueNeRF: Towards Realistic Avatar Face-to-Face Conversation Video Generationcited by 30
  32. 2023FaceXHuBERT: Text-less Speech-driven E(X)pressive 3D Facial Animation Synthesis Using Self-Supervised Speech Representation Learningcited by 29
  33. 2024UniTalker: Scaling up Audio-Driven 3D Facial Animation through A Unified Modelcited by 29
  34. 2023Speech Driven Video Editing via an Audio-Conditioned Diffusion Modelcited by 28
  35. 2024MultiTalk: Enhancing 3D Talking Head Generation Across Languages with Multilingual Video Datasetcited by 28
  36. 2025ARTalk: Speech-Driven 3D Head Animation via Autoregressive Modelcited by 28
  37. 2023LaughTalk: Expressive 3D Talking Head Generation with Laughtercited by 27
  38. 2023AI-Generated Content (AIGC) for Various Data Modalities: A Surveycited by 25

50 most cited of 257 All 257 (JSON)