mlsanityData to 7 Oct 2026

Talking Face Generation by Adversarially Disentangled Audio-Visual Representation

arXiv 1807.07860 ↗First posted 20 Jul 2018cs.CV cs.LGcited by 212

Not on the map. Why?

Abstract (arXiv)

Talking face generation aims to synthesize a sequence of face images that correspond to a clip of speech. This is a challenging task because face appearance variation and semantics of speech are coupled together in the subtle movements of the talking face regions. Existing works either construct specific face appearance model on specific subjects or model the transformation between lip motion and speech. In this work, we integrate both aspects and enable arbitrary-subject talking face generation by learning disentangled audio-visual representation. We find that the talking face sequence is actually a composition of both subject-related information and speech-related information. These two spaces are then explicitly disentangled through a novel associative-and-adversarial training process. This disentangled representation has an advantage where both audio and video can serve as inputs for generation. Extensive experiments show that the proposed approach generates realistic talking face sequences on arbitrary subjects with much clearer lip motion patterns than previous work. We also demonstrate the learned audio-visual representation is extremely useful for the tasks of automatic lip reading and audio-video retrieval.

Key facts. Cited by 212 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 Jul 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 38; 2022: 47; 2023: 56; 2024: 37; 2025: 24; 2026: 10.

212 papers.

Sort:

By area:

  1. 2021AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesiscited by 339
  2. 2021Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representationcited by 288
  3. 2022CelebV-HQ: A Large-Scale Video Facial Attributes Datasetcited by 246
  4. 2021FaceFormer: Speech-Driven 3D Facial Animation with Transformerscited by 208
  5. 2021PIRenderer: Controllable Portrait Image Generation via Semantic Neural Renderingcited by 202
  6. 2021MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglementcited by 189
  7. 2021Audio-Driven Emotional Video Portraitscited by 180
  8. 2022Depth-Aware Generative Adversarial Network for Talking Head Video Generationcited by 177
  9. 2022StyleHEAT: One-Shot High-Resolution Editable Talking Face Generation via Pre-trained StyleGANcited by 169
  10. 2022EAMM: One-Shot Emotional Talking Face via Audio-Based Emotion-Aware Motion Modelcited by 151
  11. 2021Audio2Head: Audio-driven One-shot Talking-head Generation with Natural Head Motioncited by 148
  12. 2023GeneFace: Generalized and High-Fidelity Audio-Driven 3D Talking Face Synthesiscited by 131
38 more
  1. 2022Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generationcited by 119
  2. 2021VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistencycited by 114
  3. 2022Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generationcited by 106
  4. 2022VideoReTalking: Audio-based Lip Synchronization for Talking Head Video Editing In the Wildcited by 106
  5. 2023Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expertcited by 98
  6. 2022Learning Dynamic Facial Radiance Fields for Few-Shot Talking Head Synthesiscited by 97
  7. 2023StyleTalk: One-shot Talking Head Generation with Controllable Speaking Stylescited by 92
  8. 2021FACIAL: Synthesizing Dynamic Talking Face with Implicit Attribute Learningcited by 90
  9. 2023StyleSync: High-Fidelity Generalized and Personalized Lip Sync in Style-based Generatorcited by 89
  10. 2021One-shot Talking Face Generation from Single-speaker Audio-Visual Correlation Learningcited by 87
  11. 2023SyncTalk: The Devil is in the Synchronization for Talking Head Synthesiscited by 86
  12. 2023A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need?cited by 84
  13. 2021Deepfakes Generation and Detection: State-of-the-art, open challenges, countermeasures, and way forwardcited by 82
  14. 2022Real-time Neural Radiance Talking Portrait Synthesis via Audio-spatial Decompositioncited by 81
  15. 2023Efficient Emotional Adaptation for Audio-Driven Talking-Head Generationcited by 81
  16. 2022DFA-NeRF: Personalized Talking Head Generation via Disentangled Face Attributes Neural Renderingcited by 80
  17. 2024EDTalk: Efficient Disentanglement for Emotional Talking Head Synthesiscited by 79
  18. 2023AltFreezing for More General Video Face Forgery Detectioncited by 78
  19. 2023DINet: Deformation Inpainting Network for Realistic Face Visually Dubbing on High Resolution Videocited by 73
  20. 2021LipSync3D: Data-Efficient Learning of Personalized 3D Talking Faces from Video using Pose and Lighting Normalizationcited by 67
  21. 2023OTAvatar: One-shot Talking Face Avatar with Controllable Tri-plane Renderingcited by 67
  22. 2022Synthetic Data -- what, why and how?cited by 66
  23. 2024Deepfake Generation and Detection: A Benchmark and Surveycited by 66
  24. 2022SyncTalkFace: Talking Face Generation with Precise Lip-Syncing via Audio-Lip Memorycited by 65
  25. 2021Imitating Arbitrary Talking Style for Realistic Audio-DrivenTalking Face Synthesiscited by 56
  26. 2022Talking Head Generation with Probabilistic Audio-to-Visual Diffusion Priorscited by 55
  27. 2023FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Modelscited by 55
  28. 2021Multimodal Image Synthesis and Editing: The Generative AI Eracited by 52
  29. 2022SPACE: Speech-driven Portrait Animation with Controllable Expressioncited by 49
  30. 2024VLOGGER: Multimodal Diffusion for Embodied Avatar Synthesiscited by 48
  31. 2021Visually Informed Binaural Audio Generation without Binaural Audioscited by 46
  32. 2021Write-a-speaker: Text-based Emotional and Rhythmic Talking-head Generationcited by 46
  33. 2022Implicit Warping for Animation with Image Setscited by 45
  34. 2023RenderMe-360: A Large Digital Asset Library and Benchmarks Towards High-fidelity Head Avatarscited by 45
  35. 2022Emotion-Controllable Generalized Talking Face Generationcited by 43
  36. 2023GeneFace++: Generalized and Stable Real-Time Audio-Driven 3D Talking Face Generationcited by 42
  37. 2021Neural Dubber: Dubbing for Videos According to Scriptscited by 39
  38. 2024FlowVQTalker: High-Quality Emotional Talking Face Generation through Normalizing Flow and Quantizationcited by 39

50 most cited of 212 All 212 (JSON)