mlsanityData to 14 Oct 2026

Audio-driven Talking Face Generation with Stabilized Synchronization Loss

arXiv 2307.09368 ↗First posted 18 Jul 2023cs.CVcited by 13

Not on the map. Why?

Abstract (arXiv)

Talking face generation aims to create realistic videos with accurate lip synchronization and high visual quality, using given audio and reference video while preserving identity and visual characteristics. In this paper, we start by identifying several issues with existing synchronization learning methods. These involve unstable training, lip synchronization, and visual quality issues caused by lip-sync loss, SyncNet, and lip leaking from the identity reference. To address these issues, we first tackle the lip leaking problem by introducing a silent-lip generator, which changes the lips of the identity reference to alleviate leakage. We then introduce stabilized synchronization loss and AVSyncNet to overcome problems caused by lip-sync loss and SyncNet. Experiments show that our model outperforms state-of-the-art methods in both visual quality and lip synchronization. Comprehensive ablation studies further validate our individual contributions and their cohesive effects.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 54 of its 79 references that are in the map. First posted on arXiv 18 Jul 2023.

Builds on

54 of 79 references.

References · 54

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  6. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  7. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  8. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,849
  9. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,694
  10. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,636
  11. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 672
  12. 2017How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)cited by 525
38 more
  1. 2021AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesiscited by 339
  2. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  3. 2018Deep Audio-Visual Speech Recognitioncited by 322
  4. 2020MakeItTalk: Speaker-Aware Talking-Head Animationcited by 312
  5. 2021Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representationcited by 288
  6. 2019Hierarchical Cross-Modal Talking Face Generationwith Dynamic Pixel-Wise Losscited by 252
  7. 2019Neural Voice Puppetry: Audio-driven Facial Reenactmentcited by 250
  8. 2018Talking Face Generation by Adversarially Disentangled Audio-Visual Representationcited by 212
  9. 2020Self-Supervised Learning of Audio-Visual Objects from Videocited by 188
  10. 2021Audio-Driven Emotional Video Portraitscited by 180
  11. 2022StyleHEAT: One-Shot High-Resolution Editable Talking Face Generation via Pre-trained StyleGANcited by 169
  12. 2022EAMM: One-Shot Emotional Talking Face via Audio-Based Emotion-Aware Motion Modelcited by 151
  13. 2019Realistic Speech-Driven Facial Animation with GANscited by 142
  14. 2023Diffused Heads: Diffusion Models Beat GANs on Talking-Face Generationcited by 142
  15. 2023DiffTalk: Crafting Diffusion Models for Generalized Audio-Driven Portraits Animationcited by 136
  16. 2023GeneFace: Generalized and High-Fidelity Audio-Driven 3D Talking Face Synthesiscited by 131
  17. 2020In defence of metric learning for speaker recognitioncited by 126
  18. 2022Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generationcited by 119
  19. 2022VQFR: Blind Face Restoration with Vector-Quantized Dictionary and Parallel Decodercited by 116
  20. 2022VideoReTalking: Audio-based Lip Synchronization for Talking Head Video Editing In the Wildcited by 106
  21. 2023Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expertcited by 98
  22. 2023Identity-Preserving Talking Face Generation with Landmark and Appearance Priorscited by 98
  23. 2022Learning Dynamic Facial Radiance Fields for Few-Shot Talking Head Synthesiscited by 97
  24. 2020Towards Automatic Face-to-Face Translationcited by 96
  25. 2022Progressive Disentangled Representation Learning for Fine-Grained Controllable Talking Head Synthesiscited by 93
  26. 2021FACIAL: Synthesizing Dynamic Talking Face with Implicit Attribute Learningcited by 90
  27. 2023StyleSync: High-Fidelity Generalized and Personalized Lip Sync in Style-based Generatorcited by 89
  28. 2018Talking Face Generation by Conditional Recurrent Adversarial Networkcited by 84
  29. 2020Everybody's Talkin': Let Me Talk as You Wantcited by 83
  30. 2022Real-time Neural Radiance Talking Portrait Synthesis via Audio-spatial Decompositioncited by 81
  31. 2022DFA-NeRF: Personalized Talking Head Generation via Disentangled Face Attributes Neural Renderingcited by 80
  32. 2023DINet: Deformation Inpainting Network for Realistic Face Visually Dubbing on High Resolution Videocited by 74
  33. 2021Responsive Listening Head Generation: A Benchmark Dataset and Baselinecited by 67
  34. 2022SyncTalkFace: Talking Face Generation with Precise Lip-Syncing via Audio-Lip Memorycited by 65
  35. 2018Perfect match: Improved cross-modal embeddings for audio-visual synchronisationcited by 60
  36. 2021Imitating Arbitrary Talking Style for Realistic Audio-DrivenTalking Face Synthesiscited by 56
  37. 2021Audio-Visual Synchronisation in the wildcited by 40
  38. 2022Masked Lip-Sync Prediction by Audio-Visual Contextual Exploitation in Transformerscited by 33

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 10; 2026: 2.

13 papers.

Sort:

By area:

  1. 2025VoiceCraft-Dub: Automated Video Dubbing with Neural Codec Language Modelscited by 11
  2. 2026JUST-DUB-IT: Video Dubbing via Joint Audio-Visual Diffusioncited by 7
  3. 2025FREAK: Frequency-modulated High-fidelity and Real-time Audio-driven Talking Portrait Synthesiscited by 3
  4. 2025THEval. Evaluation Framework for Talking Head Video Generationcited by 3
  5. 2024Titanic Calling: Low Bandwidth Video Conference from the Titanic Wreckcited by 1
  6. 2025FaceEditTalker: Controllable Talking Head Generation with Facial Attribute Editingcited by 1
  7. 2025Mask-Free Audio-driven Talking Face Generation for Enhanced Visual Quality and Identity Preservationcited by 1
  8. 2025Assessing Identity Leakage in Talking Face Generation: Metrics and Evaluation Frameworkcited by 1
  9. 2025A Lightweight Pipeline for Noisy Speech Voice Cloning and Accurate Lip Sync Synthesiscited by 0
  10. 2025StableDub: Taming Diffusion Prior for Generalized and Efficient Visual Dubbingcited by 0
  11. 2025Shared Latent Representation for Joint Text-to-Audio-Visual Synthesiscited by 0
  12. 2025AI killed the video star. Audio-driven diffusion model for expressive talking head generationcited by 0
1 more
  1. 2026ComplexSync: High-Fidelity and Real-Time Lip Sync in Complex Scenarioscited by 0