mlsanityData to 7 Oct 2026

StyleLipSync: Style-based Personalized Lip-sync Video Generation

arXiv 2305.00521 ↗First posted 30 Apr 2023cs.CV cs.AI cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

In this paper, we present StyleLipSync, a style-based personalized lip-sync video generative model that can generate identity-agnostic lip-synchronizing video from arbitrary audio. To generate a video of arbitrary identities, we leverage expressive lip prior from the semantically rich latent space of a pre-trained StyleGAN, where we can also design a video consistency with a linear transformation. In contrast to the previous lip-sync methods, we introduce pose-aware masking that dynamically locates the mask to improve the naturalness over frames by utilizing a 3D parametric mesh predictor frame by frame. Moreover, we propose a few-shot lip-sync adaptation method for an arbitrary person by introducing a sync regularizer that preserves lip-sync generalization while enhancing the person-specific visual information. Extensive experiments demonstrate that our model can generate accurate lip-sync videos even with the zero-shot setting and enhance characteristics of an unseen face using a few seconds of target video through the proposed adaptation method.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 39 of its 53 references that are in the map. First posted on arXiv 30 Apr 2023.

Builds on

39 of 53 references.

References · 39

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  3. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  4. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  5. 2017Squeeze-and-Excitation Networkscited by 4,763
  6. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  7. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  8. 2020Training Generative Adversarial Networks with Limited Datacited by 1,184
  9. 2021Alias-Free Generative Adversarial Networkscited by 1,181
  10. 2018VoxCeleb2: Deep Speaker Recognitioncited by 862
  11. 2020Encoding in Style: a StyleGAN Encoder for Image-to-Image Translationcited by 727
  12. 2020First Order Motion Model for Image Animationcited by 698
27 more
  1. 2019Interpreting the Latent Space of GANs for Semantic Face Editingcited by 691
  2. 2019Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?cited by 675
  3. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 669
  4. 2020GANSpace: Discovering Interpretable GAN Controlscited by 612
  5. 2019MediaPipe: A Framework for Building Perception Pipelinescited by 572
  6. 2021Designing an Encoder for StyleGAN Image Manipulationcited by 542
  7. 2019Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Setcited by 481
  8. 2021Pivotal Tuning for Latent-based Editing of Real Imagescited by 427
  9. 2020Closed-Form Factorization of Latent Semantics in GANscited by 407
  10. 2019Image2StyleGAN++: How to Edit the Embedded Images?cited by 356
  11. 2021Towards Real-World Blind Face Restoration with Generative Facial Priorcited by 347
  12. 2020MakeItTalk: Speaker-Aware Talking-Head Animationcited by 310
  13. 2021Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representationcited by 288
  14. 2019Hierarchical Cross-Modal Talking Face Generationwith Dynamic Pixel-Wise Losscited by 252
  15. 2019Neural Voice Puppetry: Audio-driven Facial Reenactmentcited by 249
  16. 2021Few-shot Image Generation via Cross-domain Correspondencecited by 206
  17. 2021GAN Prior Embedded Network for Blind Face Restoration in the Wildcited by 197
  18. 2022Latent Image Animator: Learning to Animate Images via Latent Space Navigationcited by 172
  19. 2021A Good Image Generator Is What You Need for High-Resolution Video Synthesiscited by 162
  20. 2022VideoReTalking: Audio-based Lip Synchronization for Talking Head Video Editing In the Wildcited by 106
  21. 2020Towards Automatic Face-to-Face Translationcited by 95
  22. 2022Stitch it in Time: GAN-Based Facial Editing of Real Videoscited by 68
  23. 2022SyncTalkFace: Talking Face Generation with Precise Lip-Syncing via Audio-Lip Memorycited by 65
  24. 2022Third Time's the Charm? Image and Video Editing with StyleGAN3cited by 54
  25. 2022GCFSR: a Generative and Controllable Face Super Resolution Method Without Facial and GAN Priorscited by 39
  26. 2022StyleSwap: Style-Based Generator Empowers Robust Face Swappingcited by 39
  27. 2022StyleTalker: One-shot Style-based Audio-driven Talking Head Video Generationcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 4; 2025: 4; 2026: 3.

12 papers.

  1. 2024FLOAT: Generative Motion Latent Flow Matching for Audio-driven Talking Portraitcited by 40
  2. 2024Lips Are Lying: Spotting the Temporal Inconsistency between Audio and Visual in Lip-Syncing DeepFakescited by 32
  3. 2026Avatar Forcing: Real-Time Interactive Head Avatar Generation for Natural Conversationcited by 17
  4. 2024Learning to Generate Conditional Tri-plane for 3D-aware Expression Controllable Portrait Animationcited by 15
  5. 2025KeySync: A Robust Approach for Leakage-free Lip Synchronization in High Resolutioncited by 11
  6. 2025Playmate: Flexible Control of Portrait Animation via 3D-Implicit Space Guided Diffusioncited by 8
  7. 2025FlowDubber: Movie Dubbing with LLM-based Semantic-aware Learning and Flow Matching based Voice Enhancingcited by 6
  8. 2023GMTalker: Gaussian Mixture-based Audio-Driven Emotional Talking Video Portraitscited by 2
  9. 2024SwapTalk: Audio-Driven Talking Face Generation with One-Shot Customization in Latent Spacecited by 0
  10. 2025FlashLips: 100-FPS Mask-Free Latent Lip-Sync using Reconstruction Instead of Diffusion or GANscited by 0
  11. 2026HighSync: High-Quality Lip Synchronization via Latent Diffusion Modelscited by 0
  12. 2026ComplexSync: High-Fidelity and Real-Time Lip Sync in Complex Scenarioscited by 0