mlsanityData to 14 Oct 2026

Learn2Talk: 3D Talking Face Learns from 2D Talking Face

arXiv 2404.12888 ↗First posted 19 Apr 2024cs.CV cs.GR cs.LGcited by 16

Not on the map. Why?

Abstract (arXiv)

Speech-driven facial animation methods usually contain two main classes, 3D and 2D talking face, both of which attract considerable research attention in recent years. However, to the best of our knowledge, the research on 3D talking face does not go deeper as 2D talking face, in the aspect of lip-synchronization (lip-sync) and speech perception. To mind the gap between the two sub-fields, we propose a learning framework named Learn2Talk, which can construct a better 3D talking face network by exploiting two expertise points from the field of 2D talking face. Firstly, inspired by the audio-video sync network, a 3D sync-lip expert model is devised for the pursuit of lip-sync between audio and 3D facial motion. Secondly, a teacher model selected from 2D talking face methods is used to guide the training of the audio-to-3D motions regression network to yield more 3D vertex accuracy. Extensive experiments show the advantages of the proposed framework in terms of lip-sync, vertex accuracy and speech perception, compared with state-of-the-arts. Finally, we show two applications of the proposed framework: audio-visual speech recognition and speech-driven 3D Gaussian Splatting based avatar animation.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 51 of its 80 references that are in the map. First posted on arXiv 19 Apr 2024.

Builds on

51 of 80 references.

References · 51

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
  5. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  6. 2017Neural Discrete Representation Learningcited by 4,712
  7. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  8. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  9. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 672
  10. 2019Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Setcited by 482
  11. 2020Learning an Animatable Detailed 3D Face Model from In-The-Wild Imagescited by 477
  12. 2022SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animationcited by 382
38 more
  1. 2020MakeItTalk: Speaker-Aware Talking-Head Animationcited by 312
  2. 2021Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representationcited by 288
  3. 2023GaussianAvatars: Photorealistic Head Avatars with Rigged 3D Gaussianscited by 260
  4. 2019Capture, Learning, and Synthesis of 3D Speaking Stylescited by 259
  5. 2022EMOCA: Emotion Driven Monocular Face Capture and Animationcited by 255
  6. 2019Hierarchical Cross-Modal Talking Face Generationwith Dynamic Pixel-Wise Losscited by 252
  7. 2024A Survey on 3D Gaussian Splattingcited by 224
  8. 2021FaceFormer: Speech-Driven 3D Facial Animation with Transformerscited by 209
  9. 2021MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglementcited by 190
  10. 2022Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Modelscited by 190
  11. 2023CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Priorcited by 184
  12. 2022Generating Holistic 3D Human Motion from Speechcited by 180
  13. 2019Learning to Regress 3D Face Shape and Expression from an Image without 3D Supervisioncited by 176
  14. 2022TEACH: Temporal Action Composition for 3D Humanscited by 176
  15. 2018Lip Movements Generation at a Glancecited by 155
  16. 2019Realistic Speech-Driven Facial Animation with GANscited by 142
  17. 2023EmoTalk: Speech-Driven Emotional Disentanglement for 3D Face Animationcited by 142
  18. 2021SimPoE: Simulated Character Control for 3D Human Pose Estimationcited by 114
  19. 2023Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labelscited by 114
  20. 2020Talking-head Generation with Rhythmic Head Motioncited by 112
  21. 2017You said that?cited by 99
  22. 2023Emotional Speech-Driven Animation with Content-Emotion Disentanglementcited by 98
  23. 2020Towards Automatic Face-to-Face Translationcited by 96
  24. 2022Visual Speech Recognition for Multiple Languages in the Wildcited by 95
  25. 2021One-shot Talking Face Generation from Single-speaker Audio-Visual Correlation Learningcited by 87
  26. 2023DiffPoseTalk: Speech-Driven Stylistic 3D Facial Animation and Head Pose Generation via Diffusion Modelscited by 81
  27. 2023FaceDiffuser: Speech-Driven 3D Facial Animation Synthesis Using Diffusioncited by 78
  28. 2023DINet: Deformation Inpainting Network for Realistic Face Visually Dubbing on High Resolution Videocited by 74
  29. 2022UDE: A Unified Driving Engine for Human Motion Generationcited by 67
  30. 2021Encoder-decoder with Multi-level Attention for 3D Human Shape and Pose Estimationcited by 62
  31. 2022Capturing Humans in Motion: Temporal-Attentive 3D Human Pose and Shape Estimation from Monocular Videocited by 60
  32. 2022Visual Speech-Aware Perceptual 3D Facial Expression Reconstruction from Videoscited by 50
  33. 2022Being Comes from Not-being: Open-vocabulary Text-to-Motion Generation with Wordless Trainingcited by 45
  34. 2024Media2Face: Co-speech Facial Animation Generation With Multi-Modality Guidancecited by 39
  35. 2024Towards a Simultaneous and Granular Identity-Expression Control in Personalized Face Generationcited by 26
  36. 2022Deep Learning for Visual Speech Analysis: A Surveycited by 21
  37. 2020Geometry-guided Dense Perspective Network for Speech-Driven Facial Animationcited by 19
  38. 2024UV Gaussians: Joint Learning of Mesh Deformation and Gaussian Textures for Human Avatar Modelingcited by 17

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 7; 2026: 5.

16 papers.

Sort:

By area:

  1. 2025ARTalk: Speech-Driven 3D Head Animation via Autoregressive Modelcited by 29
  2. 2025MagicInfinite: Generating Infinite Talking Videos with Your Words and Voicecited by 16
  3. 2025SkyReels-Audio: Omni Audio-Conditioned Talking Portraits in Video Diffusion Transformerscited by 14
  4. 2024MegActor-Sigma: Unlocking Flexible Mixed-Modal Control in Portrait Animation with Diffusion Transformercited by 13
  5. 2025UniLS: End-to-End Audio-Driven Avatars for Unified Listening and Speakingcited by 11
  6. 2025MemoryTalker: Personalized Speech-Driven 3D Facial Animation via Audio-Guided Stylizationcited by 9
  7. 2024GLDiTalker: Speech-Driven 3D Facial Animation with Graph Latent Diffusion Transformercited by 8
  8. 2025Advancing Talking Head Generation: A Comprehensive Survey of Multi-Modal Methodologies, Datasets, Evaluation Metrics, and Loss Functionscited by 6
  9. 2024DNPM: A Neural Parametric Model for the Synthesis of Facial Geometric Detailscited by 4
  10. 2024Diverse Code Query Learning for Speech-Driven Facial Animationcited by 2
  11. 2026CustomTex: High-fidelity Indoor Scene Texturing via Multi-Reference Customizationcited by 1
  12. 2026FMReward: Aligning and Evaluating Audio-Driven 3D Facial Animation with Human Preferencescited by 1
4 more
  1. 2025M2DAO-Talker: Harmonizing Multi-granular Motion Decoupling and Alternating Optimization for Talking-head Generationcited by 0
  2. 2026GaussianSwap: Animatable Video Face Swapping with 3D Gaussian Splattingcited by 0
  3. 2026KoUniTalk: A Lightweight Articulation-Centered Korean-English 3D Talking Face Benchmarkcited by 0
  4. 2026Seeing Speech: Learning Visible Articulatory Dynamics for Speech-Driven 3D Facial Animationcited by 0