mlsanityData to 14 Oct 2026

Perceptually Accurate 3D Talking Head Generation: New Definitions, Speech-Mesh Representation, and Evaluation Metrics

arXiv 2503.20308 ↗First posted 26 Mar 2025cs.GR cs.CVcited by 10

Not on the map. Why?

Abstract (arXiv)

Recent advancements in speech-driven 3D talking head generation have made significant progress in lip synchronization. However, existing models still struggle to capture the perceptual alignment between varying speech characteristics and corresponding lip movements. In this work, we claim that three criteria -- Temporal Synchronization, Lip Readability, and Expressiveness -- are crucial for achieving perceptually accurate lip movements. Motivated by our hypothesis that a desirable representation space exists to meet these three criteria, we introduce a speech-mesh synchronized representation that captures intricate correspondences between speech signals and 3D face meshes. We found that our learned representation exhibits desirable characteristics, and we plug it into existing models as a perceptual loss to better align lip movements to the given speech. In addition, we utilize this representation as a perceptual metric and introduce two other physically grounded lip synchronization metrics to assess how well the generated 3D talking heads align with these three criteria. Experiments show that training 3D talking head generation models with our perceptual loss significantly improve all three aspects of perceptually accurate lip synchronization. Codes and datasets are available at https://perceptual-3d-talking-head.github.io/.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 34 of its 50 references that are in the map. First posted on arXiv 26 Mar 2025.

Builds on

34 of 50 references.

References · 34

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2020Denoising Diffusion Implicit Modelscited by 9,781
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  5. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  6. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  7. 2022Human Motion Diffusion Modelcited by 1,080
  8. 2018VoxCeleb2: Deep Speaker Recognitioncited by 875
  9. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 672
  10. 2022SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animationcited by 382
  11. 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 308
  12. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 287
22 more
  1. 2019Capture, Learning, and Synthesis of 3D Speaking Stylescited by 259
  2. 2021FaceFormer: Speech-Driven 3D Facial Animation with Transformerscited by 209
  3. 2021MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglementcited by 190
  4. 2023CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Priorcited by 184
  5. 2021ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmeticcited by 150
  6. 2022Contrastive Audio-Visual Masked Autoencodercited by 146
  7. 2023EmoTalk: Speech-Driven Emotional Disentanglement for 3D Face Animationcited by 142
  8. 2023Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expertcited by 98
  9. 2023Emotional Speech-Driven Animation with Content-Emotion Disentanglementcited by 98
  10. 2024SMIRK: 3D Facial Expressions through Analysis-by-Neural-Synthesiscited by 84
  11. 2023DiffPoseTalk: Speech-Driven Stylistic 3D Facial Animation and Head Pose Generation via Diffusion Modelscited by 81
  12. 2023FaceDiffuser: Speech-Driven 3D Facial Animation Synthesis Using Diffusioncited by 78
  13. 2023SelfTalk: A Self-Supervised Commutative Training Diagram to Comprehend 3D Talking Facescited by 68
  14. 2023MuAViC: A Multilingual Audio-Visual Corpus for Robust Speech Recognition and Robust Speech-to-Text Translationcited by 39
  15. 2024HiCMAE: Hierarchical Contrastive Masked Autoencoder for Self-Supervised Audio-Visual Emotion Recognitioncited by 30
  16. 2024MultiTalk: Enhancing 3D Talking Head Generation Across Languages with Multilingual Video Datasetcited by 28
  17. 2023LaughTalk: Expressive 3D Talking Head Generation with Laughtercited by 27
  18. 2024Audio-Visual Speech Representation Expert for Enhanced Talking Face Video Generation and Evaluationcited by 17
  19. 2023Probabilistic Speech-Driven 3D Facial Motion Synthesis: New Benchmarks, Methods, and Applicationscited by 15
  20. 2023AV2AV: Direct Audio-Visual Speech to Audio-Visual Speech Translation with Unified Audio-Visual Speech Representationcited by 14
  21. 2018Phoneme-to-viseme mappings: the good, the bad, and the uglycited by 7
  22. 2024Enhancing Speech-Driven 3D Facial Animation with Audio-Visual Guidance from Lip Reading Expertcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 5.

10 papers.

  1. 2025Supervising 3D Talking Head Avatars with Analysis-by-Audio-Synthesiscited by 5
  2. 2025THEval. Evaluation Framework for Talking Head Video Generationcited by 3
  3. 2025Towards Seamless Interaction: Causal Turn-Level Modeling of Interactive 3D Conversational Head Dynamicscited by 3
  4. 2025PAVAS: Physics-Aware Video-to-Audio Synthesiscited by 2
  5. 2025VisualSpeaker: Visually-Guided 3D Avatar Lip Synthesiscited by 1
  6. 2026MANGO:Natural Multi-speaker 3D Talking Head Generation via 2D-Lifted Enhancementcited by 1
  7. 2026FMReward: Aligning and Evaluating Audio-Driven 3D Facial Animation with Human Preferencescited by 1
  8. 2026PianoFlow: Music-Aware Streaming Piano Motion Generation with Bimanual Coordinationcited by 0
  9. 2026CapTalk: Text-Guided Stylization and Speech-Driven 3D Head Animationcited by 0
  10. 2026The Shape of Speech: A Geometric Measure of Coarticulation for Speech-Driven 3D Facial Animationcited by 0