mlsanityData to 14 Oct 2026

Progressive Disentangled Representation Learning for Fine-Grained Controllable Talking Head Synthesis

arXiv 2211.14506 ↗First posted 26 Nov 2022cs.CVcited by 93

Not on the map. Why?

Cited by 22 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present a novel one-shot talking head synthesis method that achieves disentangled and fine-grained control over lip motion, eye gaze&blink, head pose, and emotional expression. We represent different motions via disentangled latent representations and leverage an image generator to synthesize talking heads from them. To effectively disentangle each motion factor, we propose a progressive disentangled representation learning strategy by separating the factors in a coarse-to-fine manner, where we first extract unified motion feature from the driving signal, and then isolate each fine-grained motion from the unified feature. We introduce motion-specific contrastive learning and regressing for non-emotional motions, and feature-level decorrelation and self-reconstruction for emotional expression, to fully utilize the inherent properties of each motion factor in unstructured video data to achieve disentanglement. Experiments show that our method provides high quality speech&lip-motion synchronization along with precise and disentangled control over multiple extra facial motions, which can hardly be achieved by previous methods.

Key facts. Cited by 93 papers in the map (data through 14 Oct 2026). Builds on 67 of its 85 references that are in the map. First posted on arXiv 26 Nov 2022.

Builds on

67 of 85 references.

References · 67

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  5. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
  6. 2018Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representationscited by 941
  7. 2018VoxCeleb2: Deep Speaker Recognitioncited by 875
  8. 2016InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Netscited by 860
  9. 2020First Order Motion Model for Image Animationcited by 704
  10. 2019Interpreting the Latent Space of GANs for Semantic Face Editingcited by 695
  11. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 672
  12. 2020Face2Face: Real-time Face Capture and Reenactment of RGB Videoscited by 632
38 more
  1. 2018Disentangling by Factorisingcited by 616
  2. 2017How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)cited by 525
  3. 2017AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wildcited by 492
  4. 2019Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Setcited by 482
  5. 2020One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencingcited by 444
  6. 2020Closed-Form Factorization of Latent Semantics in GANscited by 408
  7. 2020InterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANscited by 394
  8. 2020StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generationcited by 332
  9. 2020MakeItTalk: Speaker-Aware Talking-Head Animationcited by 312
  10. 2021Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representationcited by 288
  11. 2018Deep Video Portraitscited by 267
  12. 2022EMOCA: Emotion Driven Monocular Face Capture and Animationcited by 255
  13. 2019Hierarchical Cross-Modal Talking Face Generationwith Dynamic Pixel-Wise Losscited by 252
  14. 2019Neural Voice Puppetry: Audio-driven Facial Reenactmentcited by 250
  15. 2020Disentangled and Controllable Face Image Generation via 3D Imitative-Contrastive Learningcited by 220
  16. 2021PIRenderer: Controllable Portrait Image Generation via Semantic Neural Renderingcited by 203
  17. 2018X2Face: A network for controlling face generation by using images, audio, and pose codescited by 189
  18. 2021Audio-Driven Emotional Video Portraitscited by 180
  19. 2022Depth-Aware Generative Adversarial Network for Talking Head Video Generationcited by 178
  20. 2022Latent Image Animator: Learning to Animate Images via Latent Space Navigationcited by 175
  21. 2022StyleHEAT: One-Shot High-Resolution Editable Talking Face Generation via Pre-trained StyleGANcited by 169
  22. 2018GANimation: Anatomically-aware Facial Animation from a Single Imagecited by 161
  23. 2018Lip Movements Generation at a Glancecited by 155
  24. 2022EAMM: One-Shot Emotional Talking Face via Audio-Based Emotion-Aware Motion Modelcited by 151
  25. 2019Realistic Speech-Driven Facial Animation with GANscited by 142
  26. 2022MegaPortraits: One-shot Megapixel Neural Head Avatarscited by 131
  27. 2020Talking-head Generation with Rhythmic Head Motioncited by 112
  28. 2020Neural Head Reenactment with Latent Pose Descriptorscited by 106
  29. 2020Audio-driven Talking Face Video Generation with Learning-based Personalized Head Posecited by 92
  30. 2021FACIAL: Synthesizing Dynamic Talking Face with Implicit Attribute Learningcited by 90
  31. 2020Improved StyleGAN Embedding: Where are the Good Latents?cited by 88
  32. 2018ReenactGAN: Learning to Reenact Faces via Boundary Transfercited by 85
  33. 2020Everybody's Talkin': Let Me Talk as You Wantcited by 83
  34. 2022DFA-NeRF: Personalized Talking Head Generation via Disentangled Face Attributes Neural Renderingcited by 80
  35. 2022L2CS-Net: Fine-Grained Gaze Estimation in Unconstrained Environmentscited by 65
  36. 2020Guided Variational Autoencoder for Disentanglement Learningcited by 57
  37. 2020GIF: Generative Interpretable Facescited by 53
  38. 20223D GAN Inversion for Controllable Portrait Image Animationcited by 53

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 15; 2024: 26; 2025: 35; 2026: 16.

93 papers.

Sort:

By area:

  1. 2024Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animationcited by 202
  2. 2024VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Timecited by 187
  3. 2023HumanTOMATO: Text-aligned Whole-body Motion Generationcited by 104
  4. 2024Generalizable and Animatable Gaussian Head Avatarcited by 97
  5. 2024EDTalk: Efficient Disentanglement for Emotional Talking Head Synthesiscited by 80
  6. 2024Portrait4D-v2: Pseudo Multi-View Data Creates Better 4D Head Synthesizercited by 64
  7. 2025UniVerse-1: Unified Audio-Video Generation via Stitching of Expertscited by 61
  8. 2022Talking Head Generation with Probabilistic Audio-to-Visual Diffusion Priorscited by 55
  9. 2023Portrait4D: Learning One-Shot 4D Head Avatar Synthesis using Synthetic Datacited by 55
  10. 2023GAIA: Zero-shot Talking Avatar Generationcited by 43
  11. 2025LAM: Large Avatar Model for One-shot Animatable Gaussian Headcited by 43
  12. 2024INFP: Audio-Driven Interactive Head Generation in Dyadic Conversationscited by 40
38 more
  1. 2024FlowVQTalker: High-Quality Emotional Talking Face Generation through Normalizing Flow and Quantizationcited by 39
  2. 2025DreamActor-M1: Holistic, Expressive and Robust Human Image Animation with Hybrid Guidancecited by 36
  3. 2023TalkCLIP: Talking Head Generation with Text-Guided Expressive Speaking Stylescited by 32
  4. 2025SpeakerVid-5M: A Large-Scale High-Quality Dataset for Audio-Visual Dyadic Interactive Human Generationcited by 30
  5. 2025X-NeMo: Expressive Neural Motion Reenactment via Disentangled Latent Attentioncited by 28
  6. 2024Faces that Speak: Jointly Synthesising Talking Face and Speech from Textcited by 21
  7. 2024FaceChain-ImagineID: Freely Crafting High-Fidelity Diverse Talking Faces from Disentangled Audiocited by 20
  8. 2024VOODOO XP: Expressive One-Shot Head Reenactment for VR Telepresencecited by 19
  9. 2024TALK-Act: Enhance Textural-Awareness for 2D Speaking Avatar Reenactment with Diffusion Modelcited by 19
  10. 2025X-Streamer: Unified Human World Modeling with Audiovisual Interactioncited by 18
  11. 2024Audio-Visual Speech Representation Expert for Enhanced Talking Face Video Generation and Evaluationcited by 17
  12. 2023MFR-Net: Multi-faceted Responsive Listening Head Generation via Denoising Diffusion Modelcited by 16
  13. 2023DreamTalk: When Emotional Talking Head Generation Meets Diffusion Probabilistic Modelscited by 16
  14. 2024InstructAvatar: Text-Guided Emotion and Motion Control for Avatar Generationcited by 16
  15. 2024EmoTalk3D: High-Fidelity Free-View Synthesis of Emotional 3D Talking Headcited by 16
  16. 2024ReSyncer: Rewiring Style-based Generator for Unified Audio-Visually Synced Facial Performercited by 16
  17. 2024EmotiveTalk: Expressive Talking Head Generation through Audio Information Decoupling and Emotional Video Diffusioncited by 16
  18. 2025Teller: Real-Time Streaming Audio-Driven Portrait Animation with Autoregressive Motion Generationcited by 16
  19. 2025AvatarArtist: Open-Domain 4D Avatarizationcited by 15
  20. 2024FSRT: Facial Scene Representation Transformer for Face Reenactment from Factorized Appearance, Head-pose, and Facial Expression Featurescited by 14
  21. 2025FixTalk: Taming Identity Leakage for High-Quality Talking Head Generation in Extreme Casescited by 14
  22. 2023Audio-driven Talking Face Generation with Stabilized Synchronization Losscited by 13
  23. 2023ExpCLIP: Bridging Text and Facial Expressions via Semantic Alignmentcited by 13
  24. 2025FantasyPortrait: Enhancing Multi-Character Portrait Animation with Expression-Augmented Diffusion Transformerscited by 13
  25. 2025X-UniMotion: Animating Human Images with Expressive, Unified and Identity-Agnostic Motion Latentscited by 13
  26. 2023AgentAvatar: Disentangling Planning, Driving and Rendering for Photorealistic Avatar Agentscited by 11
  27. 2025AV-Flow: Transforming Text to Audio-Visual Human-like Interactionscited by 9
  28. 2023That's What I Said: Fully-Controllable Talking Face Generationcited by 8
  29. 2025ConsistEdit: Highly Consistent and Precise Training-free Visual Editingcited by 6
  30. 2023FONT: Flow-guided One-shot Talking Head Generation with Natural Head Motionscited by 5
  31. 2023OSM-Net: One-to-Many One-shot Talking Head Generation with Spontaneous Head Motionscited by 5
  32. 2024A Comprehensive Taxonomy and Analysis of Talking Head Synthesis: Techniques for Portrait Generation, Driving Mechanisms, and Editingcited by 5
  33. 2024Style-Preserving Lip Sync via Audio-Aware Style Referencecited by 5
  34. 2025Training-Free Text-Guided Color Editing with Multi-Modal Diffusion Transformercited by 5
  35. 2025EDTalk++: Full Disentanglement for Controllable Talking Head Synthesiscited by 5
  36. 2025FlashPortrait: 6x Faster Infinite Portrait Animation with Adaptive Latent Predictioncited by 5
  37. 2025DyStream: Streaming Dyadic Talking Heads Generation via Flow Matching-based Autoregressive Modelcited by 5
  38. 2026UIKA: Fast Universal Head Avatar from Pose-Free Imagescited by 5

50 most cited of 93 All 93 (JSON)