mlsanityData to 7 Oct 2026

AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis

arXiv 2103.11078 ↗First posted 20 Mar 2021cs.CVcited by 339

Not on the map. Why?

Cited by 38 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Generating high-fidelity talking head video by fitting with the input audio sequence is a challenging problem that receives considerable attentions recently. In this paper, we address this problem with the aid of neural scene representation networks. Our method is completely different from existing methods that rely on intermediate representations like 2D landmarks or 3D face models to bridge the gap between audio input and video output. Specifically, the feature of input audio signal is directly fed into a conditional implicit function to generate a dynamic neural radiance field, from which a high-fidelity talking-head video corresponding to the audio signal is synthesized using volume rendering. Another advantage of our framework is that not only the head (with hair) region is synthesized as previous methods did, but also the upper body is generated via two individual neural radiance fields. Experimental results demonstrate that our novel framework can (1) produce high-fidelity and natural results, and (2) support free adjustment of audio signals, viewing directions, and background images. Code is available at https://github.com/YudongGuo/AD-NeRF.

Key facts. Cited by 339 papers in the map (data through 7 Oct 2026). Builds on 30 of its 63 references that are in the map. First posted on arXiv 20 Mar 2021.

Builds on

30 of 63 references.

References · 30

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2020D-NeRF: Neural Radiance Fields for Dynamic Scenescited by 1,459
  4. 2020NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collectionscited by 1,230
  5. 2019Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representationscited by 864
  6. 2019PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitizationcited by 818
  7. 2020Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humanscited by 634
  8. 2020Face2Face: Real-time Face Capture and Reenactment of RGB Videoscited by 631
  9. 2019MaskGAN: Towards Diverse and Interactive Facial Image Manipulationcited by 624
  10. 2019Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Setcited by 481
  11. 2020Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstructioncited by 465
  12. 2015Deep Speech 2: End-to-End Speech Recognition in English and Mandarincited by 449
18 more
  1. 2020One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencingcited by 438
  2. 2019Few-Shot Adversarial Learning of Realistic Neural Talking Head Modelscited by 284
  3. 2018Deep Video Portraitscited by 265
  4. 2019Capture, Learning, and Synthesis of 3D Speaking Stylescited by 257
  5. 2019Hierarchical Cross-Modal Talking Face Generationwith Dynamic Pixel-Wise Losscited by 252
  6. 2019Neural Voice Puppetry: Audio-driven Facial Reenactmentcited by 249
  7. 2018Talking Face Generation by Adversarially Disentangled Audio-Visual Representationcited by 212
  8. 2018X2Face: A network for controlling face generation by using images, audio, and pose codescited by 189
  9. 2021NeRF--: Neural Radiance Fields Without Known Camera Parameterscited by 155
  10. 2019Realistic Speech-Driven Facial Animation with GANscited by 142
  11. 2019Text-based Editing of Talking-head Videocited by 102
  12. 2017You said that?cited by 99
  13. 2020Learning Compositional Radiance Fields of Dynamic Human Headscited by 78
  14. 2020Portrait Neural Radiance Fields from a Single Imagecited by 61
  15. 2019Neural Style-Preserving Visual Dubbingcited by 47
  16. 2021PVA: Pixel-aligned Volumetric Avatarscited by 40
  17. 2020Neural Human Video Rendering by Learning Dynamic Textures and Rendering-to-Video Translationcited by 13
  18. 2020Lightweight Photometric Stereo for Facial Details Recoverycited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 10; 2022: 48; 2023: 102; 2024: 86; 2025: 58; 2026: 35.

339 papers.

Sort:

By area:

  1. 2021Neural Fields in Visual Computing and Beyondcited by 539
  2. 2022SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animationcited by 380
  3. 2021Advances in Neural Renderingcited by 365
  4. 2022CelebV-HQ: A Large-Scale Video Facial Attributes Datasetcited by 246
  5. 2021HeadNeRF: A Real-time NeRF-based Parametric Head Modelcited by 228
  6. 2021Neural Head Avatars from Monocular RGB Videoscited by 212
  7. 2024VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Timecited by 186
  8. 2023CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Priorcited by 183
  9. 2022StyleHEAT: One-Shot High-Resolution Editable Talking Face Generation via Pre-trained StyleGANcited by 169
  10. 2022NeRF: Neural Radiance Field in 3D Vision: A Comprehensive Review (Updated Post-Gaussian Splatting)cited by 156
  11. 2023Diffused Heads: Diffusion Models Beat GANs on Talking-Face Generationcited by 142
  12. 2023EmoTalk: Speech-Driven Emotional Disentanglement for 3D Face Animationcited by 141
38 more
  1. 2023DiffTalk: Crafting Diffusion Models for Generalized Audio-Driven Portraits Animationcited by 136
  2. 2023GeneFace: Generalized and High-Fidelity Audio-Driven 3D Talking Face Synthesiscited by 131
  3. 2022Reconstructing Personalized Semantic Facial NeRF Models From Monocular Videocited by 128
  4. 2025OmniHuman-1: Rethinking the Scaling-Up of One-Stage Conditioned Human Animation Modelscited by 127
  5. 2022Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generationcited by 119
  6. 2021FENeRF: Face Editing in Neural Radiance Fieldscited by 112
  7. 2023FlashAvatar: High-fidelity Head Avatar with Efficient Gaussian Embeddingcited by 107
  8. 2022VideoReTalking: Audio-based Lip Synchronization for Talking Head Video Editing In the Wildcited by 106
  9. 2024FreGS: 3D Gaussian Splatting with Progressive Frequency Regularizationcited by 103
  10. 2022Generative Neural Articulated Radiance Fieldscited by 101
  11. 2022Learning Dynamic Facial Radiance Fields for Few-Shot Talking Head Synthesiscited by 97
  12. 2023Identity-Preserving Talking Face Generation with Landmark and Appearance Priorscited by 97
  13. 2024Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesiscited by 97
  14. 2024Generalizable and Animatable Gaussian Head Avatarcited by 95
  15. 2023PanoHead: Geometry-Aware 3D Full-Head Synthesis in 360circcited by 93
  16. 2023StyleTalk: One-shot Talking Head Generation with Controllable Speaking Stylescited by 92
  17. 2023StyleSync: High-Fidelity Generalized and Personalized Lip Sync in Style-based Generatorcited by 89
  18. 2023MonoGaussianAvatar: Monocular Gaussian Point-based Head Avatarcited by 89
  19. 2024X-Portrait: Expressive Portrait Animation with Hierarchical Motion Attentioncited by 88
  20. 2021One-shot Talking Face Generation from Single-speaker Audio-Visual Correlation Learningcited by 87
  21. 2023SyncTalk: The Devil is in the Synchronization for Talking Head Synthesiscited by 86
  22. 2023AvatarReX: Real-time Expressive Full-body Avatarscited by 83
  23. 2022Real-time Neural Radiance Talking Portrait Synthesis via Audio-spatial Decompositioncited by 81
  24. 2023Efficient Emotional Adaptation for Audio-Driven Talking-Head Generationcited by 81
  25. 2022DFA-NeRF: Personalized Talking Head Generation via Disentangled Face Attributes Neural Renderingcited by 80
  26. 2023AV-Deepfake1M: A Large-Scale LLM-Driven Audio-Visual Deepfake Datasetcited by 79
  27. 2024GPAvatar: Generalizable and Precise Head Avatar from Image(s)cited by 74
  28. 2022AvatarMAV: Fast 3D Head Avatar Reconstruction Using Motion-Aware Neural Voxelscited by 73
  29. 2022Imitator: Personalized Speech-driven 3D Facial Animationcited by 73
  30. 2023DINet: Deformation Inpainting Network for Realistic Face Visually Dubbing on High Resolution Videocited by 73
  31. 2023Efficient Region-Aware Neural Radiance Fields for High-Fidelity Talking Portrait Synthesiscited by 72
  32. 2023One-Shot High-Fidelity Talking-Head Synthesis with Deformable Neural Radiance Fieldcited by 68
  33. 2023OTAvatar: One-shot Talking Face Avatar with Controllable Tri-plane Renderingcited by 67
  34. 2023SelfTalk: A Self-Supervised Commutative Training Diagram to Comprehend 3D Talking Facescited by 67
  35. 2024Deepfake Generation and Detection: A Benchmark and Surveycited by 66
  36. 2024Advances in 3D Generation: A Surveycited by 62
  37. 2022Interactive Segmentation of Radiance Fieldscited by 60
  38. 2024TalkingGaussian: Structure-Persistent 3D Talking Head Synthesis via Gaussian Splattingcited by 60

50 most cited of 339 All 339 (JSON)