mlsanityData to 7 Oct 2026

Audio-Visual Speech Representation Expert for Enhanced Talking Face Video Generation and Evaluation

arXiv 2405.04327 ↗First posted 7 May 2024cs.CVcited by 17

Not on the map. Why?

Abstract (arXiv)

In the task of talking face generation, the objective is to generate a face video with lips synchronized to the corresponding audio while preserving visual details and identity information. Current methods face the challenge of learning accurate lip synchronization while avoiding detrimental effects on visual quality, as well as robustly evaluating such synchronization. To tackle these problems, we propose utilizing an audio-visual speech representation expert (AV-HuBERT) for calculating lip synchronization loss during training. Moreover, leveraging AV-HuBERT's features, we introduce three novel lip synchronization evaluation metrics, aiming to provide a comprehensive assessment of lip synchronization performance. Experimental results, along with a detailed ablation study, demonstrate the effectiveness of our approach and the utility of the proposed evaluation metrics.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 38 of its 64 references that are in the map. First posted on arXiv 7 May 2024.

Builds on

38 of 64 references.

References · 38

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  5. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,678
  6. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  7. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 669
  8. 2017How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)cited by 524
  9. 2021Towards Real-World Blind Face Restoration with Generative Facial Priorcited by 347
  10. 2021AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesiscited by 339
  11. 2018Deep Audio-Visual Speech Recognitioncited by 318
  12. 2020MakeItTalk: Speaker-Aware Talking-Head Animationcited by 310
26 more
  1. 2021Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representationcited by 288
  2. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 277
  3. 2019Hierarchical Cross-Modal Talking Face Generationwith Dynamic Pixel-Wise Losscited by 252
  4. 2019Neural Voice Puppetry: Audio-driven Facial Reenactmentcited by 249
  5. 2018Talking Face Generation by Adversarially Disentangled Audio-Visual Representationcited by 212
  6. 2022StyleHEAT: One-Shot High-Resolution Editable Talking Face Generation via Pre-trained StyleGANcited by 169
  7. 2018Lip Movements Generation at a Glancecited by 154
  8. 2023GeneFace: Generalized and High-Fidelity Audio-Driven 3D Talking Face Synthesiscited by 131
  9. 2020In defence of metric learning for speaker recognitioncited by 123
  10. 2022Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generationcited by 119
  11. 2022VideoReTalking: Audio-based Lip Synchronization for Talking Head Video Editing In the Wildcited by 106
  12. 2023Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expertcited by 98
  13. 2022Learning Dynamic Facial Radiance Fields for Few-Shot Talking Head Synthesiscited by 97
  14. 2023Identity-Preserving Talking Face Generation with Landmark and Appearance Priorscited by 97
  15. 2022Progressive Disentangled Representation Learning for Fine-Grained Controllable Talking Head Synthesiscited by 93
  16. 2021FACIAL: Synthesizing Dynamic Talking Face with Implicit Attribute Learningcited by 90
  17. 2020Everybody's Talkin': Let Me Talk as You Wantcited by 83
  18. 2022Real-time Neural Radiance Talking Portrait Synthesis via Audio-spatial Decompositioncited by 81
  19. 2022DFA-NeRF: Personalized Talking Head Generation via Disentangled Face Attributes Neural Renderingcited by 80
  20. 2023DINet: Deformation Inpainting Network for Realistic Face Visually Dubbing on High Resolution Videocited by 73
  21. 2022Robust Self-Supervised Audio-Visual Speech Recognitioncited by 72
  22. 2022SyncTalkFace: Talking Face Generation with Precise Lip-Syncing via Audio-Lip Memorycited by 65
  23. 2021Imitating Arbitrary Talking Style for Realistic Audio-DrivenTalking Face Synthesiscited by 56
  24. 2021Multimodal Image Synthesis and Editing: The Generative AI Eracited by 52
  25. 2021Neural Emotion Director: Speech-preserving semantic control of facial expressions in "in-the-wild" videoscited by 28
  26. 2022Face-Dubbing++: Lip-Synchronous, Voice Preserving Translation of Videoscited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 13; 2026: 3.

17 papers.

Sort:

By area:

  1. 2025PIER: A Novel Metric for Evaluating What Matters in Code-Switchingcited by 11
  2. 2025KeyFace: Expressive Audio-Driven Facial Animation for Long Sequences via KeyFrame Interpolationcited by 10
  3. 2025VoiceCraft-Dub: Automated Video Dubbing with Neural Codec Language Modelscited by 10
  4. 2025Perceptually Accurate 3D Talking Head Generation: New Definitions, Speech-Mesh Representation, and Evaluation Metricscited by 9
  5. 2025AlignDiT: Multimodal Aligned Diffusion Transformer for Synchronized Speech Generationcited by 6
  6. 2025FlowDubber: Movie Dubbing with LLM-based Semantic-aware Learning and Flow Matching based Voice Enhancingcited by 6
  7. 2025JAM-Flow: Joint Audio-Motion Synthesis with Flow Matchingcited by 3
  8. 2024Titanic Calling: Low Bandwidth Video Conference from the Titanic Wreckcited by 1
  9. 2025Mask-Free Audio-driven Talking Face Generation for Enhanced Visual Quality and Identity Preservationcited by 1
  10. 2025THEval. Evaluation Framework for Talking Head Video Generationcited by 1
  11. 2026TBDub: Production-Oriented Visual Dubbingcited by 1
  12. 2025Audio-Visual Driven Compression for Low-Bitrate Talking Head Videoscited by 0
5 more
  1. 2025SyncLipMAE: Contrastive Masked Pretraining for Audio-Visual Talking-Face Representationcited by 0
  2. 2025Shared Latent Representation for Joint Text-to-Audio-Visual Synthesiscited by 0
  3. 2025Assessing Identity Leakage in Talking Face Generation: Metrics and Evaluation Frameworkcited by 0
  4. 2026CoSyncDiT: Cognitive Synchronous Diffusion Transformer for Movie Dubbingcited by 0
  5. 2026ComplexSync: High-Fidelity and Real-Time Lip Sync in Complex Scenarioscited by 0