mlsanityData to 7 Oct 2026

StyleSync: High-Fidelity Generalized and Personalized Lip Sync in Style-based Generator

arXiv 2305.05445 ↗First posted 9 May 2023cs.CV cs.GR cs.MMcited by 89

Not on the map. Why?

Cited by 27 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Despite recent advances in syncing lip movements with any audio waves, current methods still struggle to balance generation quality and the model's generalization ability. Previous studies either require long-term data for training or produce a similar movement pattern on all subjects with low quality. In this paper, we propose StyleSync, an effective framework that enables high-fidelity lip synchronization. We identify that a style-based generator would sufficiently enable such a charming property on both one-shot and few-shot scenarios. Specifically, we design a mask-guided spatial information encoding module that preserves the details of the given face. The mouth shapes are accurately modified by audio through modulated convolutions. Moreover, our design also enables personalized lip-sync by introducing style space and generator refinement on only limited frames. Thus the identity and talking style of a target person could be accurately preserved. Extensive experiments demonstrate the effectiveness of our method in producing high-fidelity results on a variety of scenes. Resources can be found at https://hangz-nju-cuhk.github.io/projects/StyleSync.

Key facts. Cited by 89 papers in the map (data through 7 Oct 2026). Builds on 56 of its 70 references that are in the map. First posted on arXiv 9 May 2023.

Builds on

56 of 70 references.

References · 56

  1. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  2. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  3. 2021Alias-Free Generative Adversarial Networkscited by 1,181
  4. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,176
  5. 2018VoxCeleb2: Deep Speaker Recognitioncited by 862
  6. 2020Encoding in Style: a StyleGAN Encoder for Image-to-Image Translationcited by 727
  7. 2017VoxCeleb: a large-scale speaker identification datasetcited by 714
  8. 2019Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?cited by 675
  9. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 669
  10. 2021Designing an Encoder for StyleGAN Image Manipulationcited by 542
  11. 2019Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Setcited by 481
  12. 2021Pivotal Tuning for Latent-based Editing of Real Imagescited by 427
38 more
  1. 2020Closed-Form Factorization of Latent Semantics in GANscited by 407
  2. 2020InterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANscited by 393
  3. 2019Image2StyleGAN++: How to Edit the Embedded Images?cited by 356
  4. 2021Towards Real-World Blind Face Restoration with Generative Facial Priorcited by 347
  5. 2021AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesiscited by 339
  6. 2020MakeItTalk: Speaker-Aware Talking-Head Animationcited by 310
  7. 2021Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representationcited by 288
  8. 2019Few-Shot Adversarial Learning of Realistic Neural Talking Head Modelscited by 284
  9. 2019Hierarchical Cross-Modal Talking Face Generationwith Dynamic Pixel-Wise Losscited by 252
  10. 2019Neural Voice Puppetry: Audio-driven Facial Reenactmentcited by 249
  11. 2021ReStyle: A Residual-Based StyleGAN Encoder via Iterative Refinementcited by 248
  12. 2018Talking Face Generation by Adversarially Disentangled Audio-Visual Representationcited by 212
  13. 2021HyperStyle: StyleGAN Inversion with HyperNetworks for Real Image Editingcited by 208
  14. 2021FaceFormer: Speech-Driven 3D Facial Animation with Transformerscited by 208
  15. 2021GAN Prior Embedded Network for Blind Face Restoration in the Wildcited by 197
  16. 2021MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglementcited by 189
  17. 2021Audio-Driven Emotional Video Portraitscited by 180
  18. 2022StyleHEAT: One-Shot High-Resolution Editable Talking Face Generation via Pre-trained StyleGANcited by 169
  19. 2018Lip Movements Generation at a Glancecited by 154
  20. 2022EAMM: One-Shot Emotional Talking Face via Audio-Based Emotion-Aware Motion Modelcited by 151
  21. 2021Audio2Head: Audio-driven One-shot Talking-head Generation with Natural Head Motioncited by 148
  22. 2023GeneFace: Generalized and High-Fidelity Audio-Driven 3D Talking Face Synthesiscited by 131
  23. 2022Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generationcited by 119
  24. 2020Talking-head Generation with Rhythmic Head Motioncited by 112
  25. 2020Neural Head Reenactment with Latent Pose Descriptorscited by 106
  26. 2022Learning Dynamic Facial Radiance Fields for Few-Shot Talking Head Synthesiscited by 97
  27. 2020Audio-driven Talking Face Video Generation with Learning-based Personalized Head Posecited by 92
  28. 2023StyleTalk: One-shot Talking Head Generation with Controllable Speaking Stylescited by 92
  29. 2021FACIAL: Synthesizing Dynamic Talking Face with Implicit Attribute Learningcited by 90
  30. 2021One-shot Talking Face Generation from Single-speaker Audio-Visual Correlation Learningcited by 87
  31. 2018Talking Face Generation by Conditional Recurrent Adversarial Networkcited by 84
  32. 2020Everybody's Talkin': Let Me Talk as You Wantcited by 83
  33. 2022Real-time Neural Radiance Talking Portrait Synthesis via Audio-spatial Decompositioncited by 81
  34. 2023DINet: Deformation Inpainting Network for Realistic Face Visually Dubbing on High Resolution Videocited by 73
  35. 2021LipSync3D: Data-Efficient Learning of Personalized 3D Talking Faces from Video using Pose and Lighting Normalizationcited by 67
  36. 2022SyncTalkFace: Talking Face Generation with Precise Lip-Syncing via Audio-Lip Memorycited by 65
  37. 2018VisemeNet: Audio-Driven Animator-Centric Speech Animationcited by 62
  38. 2021Imitating Arbitrary Talking Style for Realistic Audio-DrivenTalking Face Synthesiscited by 56

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 11; 2024: 24; 2025: 36; 2026: 18.

89 papers.

Sort:

By area:

  1. 2024EMO: Emote Portrait Alive -- Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditionscited by 233
  2. 2024AniPortrait: Audio-Driven Synthesis of Photorealistic Portrait Animationcited by 191
  3. 2023SyncTalk: The Devil is in the Synchronization for Talking Head Synthesiscited by 86
  4. 2025FantasyTalking: Realistic Talking Portrait Generation via Coherent Motion Synthesiscited by 77
  5. 2025Let Them Talk: Audio-Driven Multi-Person Conversational Video Generationcited by 64
  6. 2023Portrait4D: Learning One-Shot 4D Head Avatar Synthesis using Synthetic Datacited by 55
  7. 2024LatentSync: Taming Audio-Conditioned Latent Diffusion Models for Lip Sync with SyncNet Supervisioncited by 53
  8. 2024VLOGGER: Multimodal Diffusion for Embodied Avatar Synthesiscited by 48
  9. 2024HeadGAP: Few-Shot 3D Head Avatar via Generalizable Gaussian Priorscited by 44
  10. 2025StableAvatar: Infinite-Length Audio-Driven Avatar Video Generationcited by 41
  11. 2024FLOAT: Generative Motion Latent Flow Matching for Audio-driven Talking Portraitcited by 40
  12. 2024INFP: Audio-Driven Interactive Head Generation in Dyadic Conversationscited by 40
38 more
  1. 2024Lips Are Lying: Spotting the Temporal Inconsistency between Audio and Visual in Lip-Syncing DeepFakescited by 32
  2. 2024MegActor: Harness the Power of Raw Video for Vivid Portrait Animationcited by 27
  3. 2024MuseTalk: Real-Time High-Fidelity Video Dubbing via Spatio-Temporal Samplingcited by 27
  4. 2025OmniSync: Towards Universal Lip Synchronization via Diffusion Transformerscited by 26
  5. 2025Human Motion Video Generation: A Surveycited by 25
  6. 2024TALK-Act: Enhance Textural-Awareness for 2D Speaking Avatar Reenactment with Diffusion Modelcited by 19
  7. 2025Audio-visual Controlled Video Diffusion with Masked Selective State Spaces Modeling for Natural Talking Head Generationcited by 17
  8. 2026Avatar Forcing: Real-Time Interactive Head Avatar Generation for Natural Conversationcited by 17
  9. 2024ReSyncer: Rewiring Style-based Generator for Unified Audio-Visually Synced Facial Performercited by 16
  10. 2025MagicInfinite: Generating Infinite Talking Videos with Your Words and Voicecited by 16
  11. 2023Probabilistic Speech-Driven 3D Facial Motion Synthesis: New Benchmarks, Methods, and Applicationscited by 15
  12. 2024PersonaTalk: Bring Attention to Your Persona in Visual Dubbingcited by 15
  13. 2024DiffusionAct: Controllable Diffusion Autoencoder for One-shot Face Reenactmentcited by 14
  14. 2024RealTalk: Real-time and Realistic Audio-driven Face Generation with 3D Facial Prior-guided Identity Alignment Networkcited by 14
  15. 2025FixTalk: Taming Identity Leakage for High-Quality Talking Head Generation in Extreme Casescited by 14
  16. 2023Audio-driven Talking Face Generation with Stabilized Synchronization Losscited by 13
  17. 2023DreamTalk: When Emotional Talking Head Generation Meets Diffusion Probabilistic Modelscited by 13
  18. 2024MegActor-Sigma: Unlocking Flexible Mixed-Modal Control in Portrait Animation with Diffusion Transformercited by 13
  19. 2025SkyReels-Audio: Omni Audio-Conditioned Talking Portraits in Video Diffusion Transformerscited by 13
  20. 2025SayAnything: Audio-Driven Lip Synchronization with Conditional Video Diffusioncited by 12
  21. 2025KeySync: A Robust Approach for Leakage-free Lip Synchronization in High Resolutioncited by 11
  22. 2026FlowAct-R1: Towards Interactive Humanoid Video Generationcited by 11
  23. 2025AudCast: Audio-Driven Human Video Generation by Cascaded Diffusion Transformerscited by 10
  24. 2025DiTaiListener: Controllable High Fidelity Listener Video Generation with Diffusioncited by 10
  25. 2023DiffTalker: Co-driven audio-image diffusion for talking faces via intermediate landmarkscited by 9
  26. 2025OmniTalker: One-shot Real-time Text-Driven Talking Audio-Video Generation With Multimodal Style Mimickingcited by 9
  27. 2025SyncTalk++: High-Fidelity and Efficient Synchronized Talking Heads Synthesis Using Gaussian Splattingcited by 9
  28. 2024Make Your Actor Talk: Generalizable and High-Fidelity Lip Sync with Motion and Appearance Disentanglementcited by 7
  29. 2024EMOdiffhead: Continuously Emotional Control in Talking Head Generation via Diffusioncited by 6
  30. 2025Audio-Driven Universal Gaussian Head Avatarscited by 6
  31. 2026JUST-DUB-IT: Video Dubbing via Joint Audio-Visual Diffusioncited by 6
  32. 2025PC-Talk: Precise Facial Animation Control for Audio-Driven Talking Face Generationcited by 5
  33. 2025Dimitra: Audio-driven Diffusion model for Expressive Talking Head Generationcited by 4
  34. 2023ChatAnything: Facetime Chat with LLM-Enhanced Personascited by 3
  35. 2023R2-Talker: Realistic Real-Time Talking Head Synthesis with Hash Grid Landmarks Encoding and Progressive Multilayer Conditioningcited by 3
  36. 2024Dubbing for Everyone: Data-Efficient Visual Dubbing using Neural Rendering Priorscited by 3
  37. 2024A Multi-Stream Fusion Approach with One-Class Learning for Audio-Visual Deepfake Detectioncited by 3
  38. 2024JambaTalk: Speech-Driven 3D Talking Head Generation Based on Hybrid Transformer-Mamba Modelcited by 3

50 most cited of 89 All 89 (JSON)