mlsanityData to 14 Oct 2026

Instruct-NeuralTalker: Editing Audio-Driven Talking Radiance Fields with Instructions

arXiv 2306.10813 ↗First posted 19 Jun 2023cs.CV cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Recent neural talking radiance field methods have shown great success in photorealistic audio-driven talking face synthesis. In this paper, we propose a novel interactive framework that utilizes human instructions to edit such implicit neural representations to achieve real-time personalized talking face generation. Given a short speech video, we first build an efficient talking radiance field, and then apply the latest conditional diffusion model for image editing based on the given instructions and guiding implicit representation optimization towards the editing target. To ensure audio-lip synchronization during the editing process, we propose an iterative dataset updating strategy and utilize a lip-edge loss to constrain changes in the lip region. We also introduce a lightweight refinement network for complementing image details and achieving controllable detail generation in the final rendered image. Our method also enables real-time rendering at up to 30FPS on consumer hardware. Multiple metrics and user verification show that our approach provides a significant improvement in rendering quality compared to state-of-the-art methods.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 32 of its 41 references that are in the map. First posted on arXiv 19 Jun 2023.

Builds on

32 of 41 references.

References · 32

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  5. 2022Classifier-Free Diffusion Guidancecited by 5,847
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  7. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,723
  8. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,694
  9. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  10. 2014Conditional Generative Adversarial Netscited by 2,162
  11. 2020D-NeRF: Neural Radiance Fields for Dynamic Scenescited by 1,465
  12. 2022TensoRF: Tensorial Radiance Fieldscited by 1,280
20 more
  1. 2018ESRGAN: Enhanced Super-Resolution Generative Adversarial Networkscited by 1,254
  2. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,205
  3. 2017PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volumecited by 968
  4. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 672
  5. 2023Instruct-NeRF2NeRF: Editing 3D Scenes with Instructionscited by 484
  6. 2022SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animationcited by 382
  7. 2021CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fieldscited by 370
  8. 2018GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Posecited by 342
  9. 2021AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesiscited by 339
  10. 2020MakeItTalk: Speaker-Aware Talking-Head Animationcited by 312
  11. 2019Few-Shot Adversarial Learning of Realistic Neural Talking Head Modelscited by 284
  12. 2019Neural Voice Puppetry: Audio-driven Facial Reenactmentcited by 250
  13. 2017Photographic Image Synthesis with Cascaded Refinement Networkscited by 223
  14. 2022ARF: Artistic Radiance Fieldscited by 147
  15. 2022StylizedNeRF: Consistent 3D Scene Stylization as Stylized NeRF via 2D-3D Mutual Learningcited by 140
  16. 2022NeRF-Art: Text-Driven Neural Radiance Fields Stylizationcited by 126
  17. 2022Learning Dynamic Facial Radiance Fields for Few-Shot Talking Head Synthesiscited by 97
  18. 2022Real-time Neural Radiance Talking Portrait Synthesis via Audio-spatial Decompositioncited by 81
  19. 2022DFA-NeRF: Personalized Talking Head Generation via Disentangled Face Attributes Neural Renderingcited by 80
  20. 2023Blind Video Deflickering by Neural Filtering with a Flawed Atlascited by 27

Built on by

Cited by year

Citing papers per year: 2023: 1.

1 paper.

  1. 2023Efficient-NeRF2NeRF: Streamlining Text-Driven 3D Editing with Multiview Correspondence-Enhanced Diffusion Modelscited by 16

Similar, not cited

Close in wording, no citation either way

  1. 2024Resolution-Agnostic Neural Compression for High-Fidelity Portrait Video Conferencing via Implicit Radiance Fields
  2. 2024PortraitTalk: Towards Customizable One-Shot Audio-to-Talking Face Generation
  3. 2025FacEDiT: Unified Talking Face Editing and Generation via Facial Motion Infilling
  4. 2025IMTalker: Efficient Audio-driven Talking Face Generation with Implicit Motion Transfer
  5. 2025See the Speaker: Crafting High-Resolution Talking Faces from Speech with Prior Guidance and Region Refinement
  6. 2024AVI-Talking: Learning Audio-Visual Instructions for Expressive 3D Talking Face Generation
  7. 2025FantasyTalking: Realistic Talking Portrait Generation via Coherent Motion Synthesis
  8. 2025Advancing Talking Head Generation: A Comprehensive Survey of Multi-Modal Methodologies, Datasets, Evaluation Metrics, and Loss Functions