mlsanityData to 7 Oct 2026

GeneFace: Generalized and High-Fidelity Audio-Driven 3D Talking Face Synthesis

arXiv 2301.13430 ↗First posted 31 Jan 2023cs.CVcited by 131

Not on the map. Why?

Cited by 24 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Generating photo-realistic video portrait with arbitrary speech audio is a crucial problem in film-making and virtual reality. Recently, several works explore the usage of neural radiance field in this task to improve 3D realness and image fidelity. However, the generalizability of previous NeRF-based methods to out-of-domain audio is limited by the small scale of training data. In this work, we propose GeneFace, a generalized and high-fidelity NeRF-based talking face generation method, which can generate natural results corresponding to various out-of-domain audio. Specifically, we learn a variaitional motion generator on a large lip-reading corpus, and introduce a domain adaptative post-net to calibrate the result. Moreover, we learn a NeRF-based renderer conditioned on the predicted facial motion. A head-aware torso-NeRF is proposed to eliminate the head-torso separation problem. Extensive experiments show that our method achieves more generalized and high-fidelity talking face generation compared to previous methods.

Key facts. Cited by 131 papers in the map (data through 7 Oct 2026). Builds on 27 of its 40 references that are in the map. First posted on arXiv 31 Jan 2023.

Builds on

27 of 40 references.

References · 27

  1. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  2. 2020D-NeRF: Neural Radiance Fields for Dynamic Scenescited by 1,459
  3. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,204
  4. 2019Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representationscited by 864
  5. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 669
  6. 2019MaskGAN: Towards Diverse and Interactive Facial Image Manipulationcited by 624
  7. 2019Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Setcited by 481
  8. 2020Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstructioncited by 465
  9. 2014Deep Speech: Scaling up end-to-end speech recognitioncited by 354
  10. 2021AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesiscited by 339
  11. 2020MakeItTalk: Speaker-Aware Talking-Head Animationcited by 310
  12. 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 301
15 more
  1. 2021Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representationcited by 288
  2. 2019Capture, Learning, and Synthesis of 3D Speaking Stylescited by 257
  3. 2019Hierarchical Cross-Modal Talking Face Generationwith Dynamic Pixel-Wise Losscited by 252
  4. 2019Neural Voice Puppetry: Audio-driven Facial Reenactmentcited by 249
  5. 2021HeadNeRF: A Real-time NeRF-based Parametric Head Modelcited by 228
  6. 2018Talking Face Generation by Adversarially Disentangled Audio-Visual Representationcited by 212
  7. 2018X2Face: A network for controlling face generation by using images, audio, and pose codescited by 189
  8. 2018Lip Movements Generation at a Glancecited by 154
  9. 2019Realistic Speech-Driven Facial Animation with GANscited by 142
  10. 2022Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generationcited by 119
  11. 2021Neural Lumigraph Renderingcited by 105
  12. 2022Learning Dynamic Facial Radiance Fields for Few-Shot Talking Head Synthesiscited by 97
  13. 2020Audio-driven Talking Face Video Generation with Learning-based Personalized Head Posecited by 92
  14. 2022DFA-NeRF: Personalized Talking Head Generation via Disentangled Face Attributes Neural Renderingcited by 80
  15. 2021PortaSpeech: Portable and High-Quality Generative Text-to-Speechcited by 47

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 26; 2024: 45; 2025: 44; 2026: 16.

131 papers.

Sort:

By area:

  1. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 264
  2. 2024AniPortrait: Audio-Driven Synthesis of Photorealistic Portrait Animationcited by 191
  3. 2025OmniHuman-1: Rethinking the Scaling-Up of One-Stage Conditioned Human Animation Modelscited by 127
  4. 2024Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesiscited by 97
  5. 2023StyleSync: High-Fidelity Generalized and Personalized Lip Sync in Style-based Generatorcited by 89
  6. 2023SyncTalk: The Devil is in the Synchronization for Talking Head Synthesiscited by 86
  7. 2023Efficient Region-Aware Neural Radiance Fields for High-Fidelity Talking Portrait Synthesiscited by 72
  8. 2024TalkingGaussian: Structure-Persistent 3D Talking Head Synthesis via Gaussian Splattingcited by 60
  9. 2024LatentSync: Taming Audio-Conditioned Latent Diffusion Models for Lip Sync with SyncNet Supervisioncited by 53
  10. 2025Long-Term TalkingFace Generation via Motion-Prior Conditional Diffusion Modelcited by 49
  11. 2024VLOGGER: Multimodal Diffusion for Embodied Avatar Synthesiscited by 48
  12. 2023GeneFace++: Generalized and Stable Real-Time Audio-Driven 3D Talking Face Generationcited by 42
38 more
  1. 2025HumanDiT: Pose-Guided Diffusion Transformer for Long-form Human Motion Video Generationcited by 38
  2. 2024Style2Talker: High-Resolution Talking Head Generation with Emotion Style and Art Stylecited by 33
  3. 2024Audio-Synchronized Visual Animationcited by 32
  4. 2023MODA: Mapping-Once Audio-driven Portrait Animation with Dual Attentionscited by 29
  5. 2023AI-Generated Content (AIGC) for Various Data Modalities: A Surveycited by 25
  6. 2025Human Motion Video Generation: A Surveycited by 25
  7. 2023DREAM-Talk: Diffusion-based Realistic Emotional Audio-driven Method for Single Image Talking Face Generationcited by 24
  8. 2024GaussianTalker: Speaker-specific Talking Head Synthesis via 3D Gaussian Splattingcited by 24
  9. 2025StreamAvatar: Streaming Diffusion Models for Real-Time Interactive Human Avatarscited by 24
  10. 2025InterActHuman: Multi-Concept Human Animation with Layout-Aligned Audio Conditionscited by 20
  11. 2025InsTaG: Learning Personalized 3D Talking Head from Few-Second Videocited by 18
  12. 2023Speech-Driven 3D Face Animation with Composite and Regional Facial Movementscited by 17
  13. 2024GaussianTalker: Real-Time High-Fidelity Talking Head Synthesis with Audio-Driven 3D Gaussian Splattingcited by 17
  14. 2024Audio-Visual Speech Representation Expert for Enhanced Talking Face Video Generation and Evaluationcited by 17
  15. 2023AE-NeRF: Audio Enhanced Neural Radiance Field for Few Shot Talking Head Synthesiscited by 16
  16. 2024EmoTalk3D: High-Fidelity Free-View Synthesis of Emotional 3D Talking Headcited by 16
  17. 2024ReSyncer: Rewiring Style-based Generator for Unified Audio-Visually Synced Facial Performercited by 16
  18. 2024MimicTalk: Mimicking a personalized and expressive 3D talking face in minutescited by 16
  19. 2023Probabilistic Speech-Driven 3D Facial Motion Synthesis: New Benchmarks, Methods, and Applicationscited by 15
  20. 2024PersonaTalk: Bring Attention to Your Persona in Visual Dubbingcited by 15
  21. 2023ShiftDDPMs: Exploring Conditional Diffusion Models by Shifting Diffusion Trajectoriescited by 14
  22. 2024Beyond Talking -- Generating Holistic 3D Human Dyadic Motion for Communicationcited by 14
  23. 2024RealTalk: Real-time and Realistic Audio-driven Face Generation with 3D Facial Prior-guided Identity Alignment Networkcited by 14
  24. 2023Audio-driven Talking Face Generation with Stabilized Synchronization Losscited by 13
  25. 2023DreamTalk: When Emotional Talking Head Generation Meets Diffusion Probabilistic Modelscited by 13
  26. 2024Learning Dynamic Tetrahedra for High-Quality Talking Head Synthesiscited by 12
  27. 2024GSTalker: Real-time Audio-Driven Talking Face Generation via Deformable Gaussian Splattingcited by 12
  28. 2025Cafe-Talk: Generating 3D Talking Face Animation with Multimodal Coarse- and Fine-grained Controlcited by 12
  29. 2024MoDiTalker: Motion-Disentangled Diffusion Model for High-Fidelity Talking Head Generationcited by 10
  30. 2023SingingHead: A Large-scale 4D Dataset for Singing Head Animationcited by 9
  31. 2024GaussianSpeech: Audio-Driven Gaussian Avatarscited by 9
  32. 2025AV-Flow: Transforming Text to Audio-Visual Human-like Interactionscited by 9
  33. 2025SyncTalk++: High-Fidelity and Efficient Synchronized Talking Heads Synthesis Using Gaussian Splattingcited by 9
  34. 2023Ada-TTA: Towards Adaptive High-Quality Text-to-Talking Avatar Synthesiscited by 8
  35. 2023CorrTalk: Correlation Between Hierarchical Speech and Facial Activity Variances for 3D Animationcited by 8
  36. 2024NeRFFaceSpeech: One-shot Audio-driven 3D Talking Head Synthesis via Generative Priorcited by 8
  37. 2025ChatAnyone: Stylized Real-time Portrait Video Generation with Hierarchical Motion Diffusion Modelcited by 8
  38. 2023VAST: Vivify Your Talking Avatar via Zero-Shot Expressive Facial Style Transfercited by 7

50 most cited of 131 All 131 (JSON)