mlsanityData to 14 Oct 2026

JOLT3D: Joint Learning of Talking Heads and 3DMM Parameters with Application to Lip-Sync

arXiv 2507.20452 ↗First posted 28 Jul 2025cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

In this work, we revisit the effectiveness of 3DMM for talking head synthesis by jointly learning a 3D face reconstruction model and a talking head synthesis model. This enables us to obtain a FACS-based blendshape representation of facial expressions that is optimized for talking head synthesis. This contrasts with previous methods that either fit 3DMM parameters to 2D landmarks or rely on pretrained face reconstruction models. Not only does our approach increase the quality of the generated face, but it also allows us to take advantage of the blendshape representation to modify just the mouth region for the purpose of audio-based lip-sync. To this end, we propose a novel lip-sync pipeline that, unlike previous methods, decouples the original chin contour from the lip-synced chin contour, and reduces flickering near the mouth.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 68 of its 83 references that are in the map. First posted on arXiv 28 Jul 2025.

Builds on

68 of 83 references.

References · 68

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  7. 2022Classifier-Free Diffusion Guidancecited by 5,847
  8. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
  9. 2016Improved Techniques for Training GANscited by 2,970
  10. 2016Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Networkcited by 1,908
  11. 2018Group Normalizationcited by 1,524
  12. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,430
38 more
  1. 2021Palette: Image-to-Image Diffusion Modelscited by 1,300
  2. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,299
  3. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,214
  4. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,198
  5. 2018VoxCeleb2: Deep Speaker Recognitioncited by 875
  6. 2020Accelerating 3D Deep Learning with PyTorch3Dcited by 784
  7. 2020First Order Motion Model for Image Animationcited by 704
  8. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 672
  9. 2019Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Setcited by 482
  10. 2020Learning an Animatable Detailed 3D Face Model from In-The-Wild Imagescited by 477
  11. 2020One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencingcited by 444
  12. 2022SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animationcited by 382
  13. 20193D Morphable Face Models -- Past, Present and Futurecited by 270
  14. 2022EMOCA: Emotion Driven Monocular Face Capture and Animationcited by 255
  15. 2022CelebV-HQ: A Large-Scale Video Facial Attributes Datasetcited by 250
  16. 2017Geometric GANcited by 233
  17. 2023TOPIQ: A Top-down Approach from Semantics to Distortions for Image Quality Assessmentcited by 226
  18. 2024LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Controlcited by 206
  19. 2021PIRenderer: Controllable Portrait Image Generation via Semantic Neural Renderingcited by 203
  20. 2024Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animationcited by 202
  21. 2021General Facial Representation Learning in a Visual-Linguistic Mannercited by 195
  22. 2024VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Timecited by 187
  23. 2022VFHQ: A High-Quality Dataset and Benchmark for Video Face Super-Resolutioncited by 176
  24. 2017MoFA: Model-based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstructioncited by 163
  25. 2018Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separationcited by 162
  26. 2022EAMM: One-Shot Emotional Talking Face via Audio-Based Emotion-Aware Motion Modelcited by 151
  27. 2017Morphable Face Models - An Open Frameworkcited by 138
  28. 2021Robust High-Resolution Video Matting with Temporal Guidancecited by 137
  29. 2022MegaPortraits: One-shot Megapixel Neural Head Avatarscited by 131
  30. 2020HeadGAN: One-shot Neural Head Synthesis and Editingcited by 105
  31. 2024Loopy: Taming Audio-Driven Portrait Avatar with Long-Term Motion Dependencycited by 102
  32. 2023Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expertcited by 98
  33. 2023Identity-Preserving Talking Face Generation with Landmark and Appearance Priorscited by 98
  34. 2024Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesiscited by 97
  35. 2020Learning Formation of Physically-Based Face Attributescited by 94
  36. 2020Audio-driven Talking Face Video Generation with Learning-based Personalized Head Posecited by 92
  37. 2021FACIAL: Synthesizing Dynamic Talking Face with Implicit Attribute Learningcited by 90
  38. 2023AVID: Any-Length Video Inpainting with Diffusion Modelcited by 82

50 most cited of 68 All 68 (JSON)

Built on by

Not cited yet.