mlsanityData to 7 Oct 2026

Neural Voice Puppetry: Audio-driven Facial Reenactment

arXiv 1912.05566 ↗First posted 11 Dec 2019cs.CV cs.GRcited by 249

Not on the map. Why?

Abstract (arXiv)

We present Neural Voice Puppetry, a novel approach for audio-driven facial video synthesis. Given an audio sequence of a source person or digital assistant, we generate a photo-realistic output video of a target person that is in sync with the audio of the source input. This audio-driven facial reenactment is driven by a deep neural network that employs a latent 3D face model space. Through the underlying 3D representation, the model inherently learns temporal stability while we leverage neural rendering to generate photo-realistic output frames. Our approach generalizes across different people, allowing us to synthesize videos of a target actor with the voice of any unknown source actor or even synthetic voices that can be generated utilizing standard text-to-speech approaches. Neural Voice Puppetry has a variety of use-cases, including audio-driven video avatars, video dubbing, and text-driven video synthesis of a talking head. We demonstrate the capabilities of our method in a series of audio- and text-based puppetry examples, including comparisons to state-of-the-art techniques and a user study.

Key facts. Cited by 249 papers in the map (data through 7 Oct 2026). First posted on arXiv 11 Dec 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 35; 2022: 53; 2023: 76; 2024: 40; 2025: 31; 2026: 14.

249 papers.

Sort:

By area:

  1. 2022SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animationcited by 380
  2. 2021Advances in Neural Renderingcited by 365
  3. 2021AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesiscited by 339
  4. 2021Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representationcited by 288
  5. 2021Neural Head Avatars from Monocular RGB Videoscited by 212
  6. 2021FaceFormer: Speech-Driven 3D Facial Animation with Transformerscited by 208
  7. 2021PIRenderer: Controllable Portrait Image Generation via Semantic Neural Renderingcited by 202
  8. 2021MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglementcited by 189
  9. 2023CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Priorcited by 183
  10. 2021Audio-Driven Emotional Video Portraitscited by 180
  11. 2021FakeAVCeleb: A Novel Audio-Video Multimodal Deepfake Datasetcited by 179
  12. 2022Latent Image Animator: Learning to Animate Images via Latent Space Navigationcited by 172
38 more
  1. 2022StyleHEAT: One-Shot High-Resolution Editable Talking Face Generation via Pre-trained StyleGANcited by 169
  2. 2022Towards Metrical Reconstruction of Human Facescited by 165
  3. 2021Exploring Temporal Coherence for More General Video Face Forgery Detectioncited by 158
  4. 2022EAMM: One-Shot Emotional Talking Face via Audio-Based Emotion-Aware Motion Modelcited by 151
  5. 2021Audio2Head: Audio-driven One-shot Talking-head Generation with Natural Head Motioncited by 148
  6. 2023EmoTalk: Speech-Driven Emotional Disentanglement for 3D Face Animationcited by 141
  7. 2023DiffTalk: Crafting Diffusion Models for Generalized Audio-Driven Portraits Animationcited by 136
  8. 2023GeneFace: Generalized and High-Fidelity Audio-Driven 3D Talking Face Synthesiscited by 131
  9. 2022Reconstructing Personalized Semantic Facial NeRF Models From Monocular Videocited by 128
  10. 2022Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generationcited by 119
  11. 2023FlashAvatar: High-fidelity Head Avatar with Efficient Gaussian Embeddingcited by 107
  12. 2022VideoReTalking: Audio-based Lip Synchronization for Talking Head Video Editing In the Wildcited by 106
  13. 2023Emotional Speech-Driven Animation with Content-Emotion Disentanglementcited by 98
  14. 2022Learning Dynamic Facial Radiance Fields for Few-Shot Talking Head Synthesiscited by 97
  15. 2023Identity-Preserving Talking Face Generation with Landmark and Appearance Priorscited by 97
  16. 2022Progressive Disentangled Representation Learning for Fine-Grained Controllable Talking Head Synthesiscited by 93
  17. 2023StyleTalk: One-shot Talking Head Generation with Controllable Speaking Stylescited by 92
  18. 2021FACIAL: Synthesizing Dynamic Talking Face with Implicit Attribute Learningcited by 90
  19. 2023StyleSync: High-Fidelity Generalized and Personalized Lip Sync in Style-based Generatorcited by 89
  20. 2021Face Forensics in the Wildcited by 87
  21. 2021One-shot Talking Face Generation from Single-speaker Audio-Visual Correlation Learningcited by 87
  22. 2023SyncTalk: The Devil is in the Synchronization for Talking Head Synthesiscited by 86
  23. 2023A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need?cited by 84
  24. 2024From Audio to Photoreal Embodiment: Synthesizing Humans in Conversationscited by 82
  25. 2022Real-time Neural Radiance Talking Portrait Synthesis via Audio-spatial Decompositioncited by 81
  26. 2022DFA-NeRF: Personalized Talking Head Generation via Disentangled Face Attributes Neural Renderingcited by 80
  27. 2024EDTalk: Efficient Disentanglement for Emotional Talking Head Synthesiscited by 79
  28. 2023AltFreezing for More General Video Face Forgery Detectioncited by 78
  29. 2022Imitator: Personalized Speech-driven 3D Facial Animationcited by 73
  30. 2023DINet: Deformation Inpainting Network for Realistic Face Visually Dubbing on High Resolution Videocited by 73
  31. 2023Efficient Region-Aware Neural Radiance Fields for High-Fidelity Talking Portrait Synthesiscited by 72
  32. 2021LipSync3D: Data-Efficient Learning of Personalized 3D Talking Faces from Video using Pose and Lighting Normalizationcited by 67
  33. 2022SyncTalkFace: Talking Face Generation with Precise Lip-Syncing via Audio-Lip Memorycited by 65
  34. 2024TalkingGaussian: Structure-Persistent 3D Talking Head Synthesis via Gaussian Splattingcited by 60
  35. 2021Speech Drives Templates: Co-Speech Gesture Synthesis with Learned Templatescited by 57
  36. 2021Imitating Arbitrary Talking Style for Realistic Audio-DrivenTalking Face Synthesiscited by 56
  37. 2022Talking Head Generation with Probabilistic Audio-to-Visual Diffusion Priorscited by 55
  38. 2023FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Modelscited by 55

50 most cited of 249 All 249 (JSON)