mlsanityData to 7 Oct 2026

CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Prior

arXiv 2301.02379 ↗First posted 6 Jan 2023cs.CVcited by 183

Not on the map. Why?

Cited by 26 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Speech-driven 3D facial animation has been widely studied, yet there is still a gap to achieving realism and vividness due to the highly ill-posed nature and scarcity of audio-visual data. Existing works typically formulate the cross-modal mapping into a regression task, which suffers from the regression-to-mean problem leading to over-smoothed facial motions. In this paper, we propose to cast speech-driven facial animation as a code query task in a finite proxy space of the learned codebook, which effectively promotes the vividness of the generated motions by reducing the cross-modal mapping uncertainty. The codebook is learned by self-reconstruction over real facial motions and thus embedded with realistic facial motion priors. Over the discrete motion space, a temporal autoregressive model is employed to sequentially synthesize facial motions from the input speech signal, which guarantees lip-sync as well as plausible facial expressions. We demonstrate that our approach outperforms current state-of-the-art methods both qualitatively and quantitatively. Also, a user study further justifies our superiority in perceptual quality.

Key facts. Cited by 183 papers in the map (data through 7 Oct 2026). Builds on 44 of its 78 references that are in the map. First posted on arXiv 6 Jan 2023.

Builds on

44 of 78 references.

References · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  4. 2017Neural Discrete Representation Learningcited by 4,650
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  6. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  7. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  8. 2017Arbitrary Style Transfer in Real-time with Adaptive Instance Normalizationcited by 1,998
  9. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  10. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,187
  11. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  12. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 669
32 more
  1. 2021AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesiscited by 339
  2. 2021Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representationcited by 288
  3. 2018Deep Video Portraitscited by 265
  4. 2019Capture, Learning, and Synthesis of 3D Speaking Stylescited by 257
  5. 2019Neural Voice Puppetry: Audio-driven Facial Reenactmentcited by 249
  6. 2022Towards Robust Blind Face Restoration with Codebook Lookup Transformercited by 230
  7. 2017Improved Texture Networks: Maximizing Quality and Diversity in Feed-forward Stylization and Texture Synthesiscited by 215
  8. 2021FaceFormer: Speech-Driven 3D Facial Animation with Transformerscited by 208
  9. 2021MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglementcited by 189
  10. 2021Audio-Driven Emotional Video Portraitscited by 180
  11. 2018Lip Movements Generation at a Glancecited by 154
  12. 2022EAMM: One-Shot Emotional Talking Face via Audio-Based Emotion-Aware Motion Modelcited by 151
  13. 2019Realistic Speech-Driven Facial Animation with GANscited by 142
  14. 2022Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motioncited by 125
  15. 2022Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generationcited by 119
  16. 2020Talking-head Generation with Rhythmic Head Motioncited by 112
  17. 2021Generating Diverse Structure for Image Inpainting With Hierarchical VQ-VAEcited by 106
  18. 2019Text-based Editing of Talking-head Videocited by 102
  19. 2021Learning Speech-driven 3D Conversational Gestures from Videocited by 98
  20. 2022Learning Dynamic Facial Radiance Fields for Few-Shot Talking Head Synthesiscited by 97
  21. 2020Audio-driven Talking Face Video Generation with Learning-based Personalized Head Posecited by 92
  22. 2021One-shot Talking Face Generation from Single-speaker Audio-Visual Correlation Learningcited by 87
  23. 2022MetaPortrait: Identity-Preserving Talking Head Generation with Fast Personalized Adaptationcited by 69
  24. 2021LipSync3D: Data-Efficient Learning of Personalized 3D Talking Faces from Video using Pose and Lighting Normalizationcited by 67
  25. 2018VisemeNet: Audio-Driven Animator-Centric Speech Animationcited by 62
  26. 2020Audio- and Gaze-driven Facial Animation of Codec Avatarscited by 46
  27. 2022Rhythmic Gesticulator: Rhythm-Aware Co-Speech Gesture Synthesis with Hierarchical Neural Embeddingscited by 38
  28. 2022Audio-Visual Speech Codecs: Rethinking Audio-Visual Speech Enhancement by Re-Synthesiscited by 24
  29. 2022Talking Head from Speech Audio using a Pre-trained Image Generatorcited by 16
  30. 2021Joint Audio-Text Model for Expressive Speech-Driven 3D Facial Animationcited by 15
  31. 20213D-TalkEmo: Learning to Synthesize 3D Emotional Talking Headcited by 10
  32. 2022UniColor: A Unified Framework for Multi-Modal Colorization with Transformercited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 34; 2024: 49; 2025: 50; 2026: 50.

183 papers.

Sort:

By area:

  1. 2024VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Timecited by 186
  2. 2023EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modelingcited by 101
  3. 2023Emotional Speech-Driven Animation with Content-Emotion Disentanglementcited by 98
  4. 2023DiffPoseTalk: Speech-Driven Stylistic 3D Facial Animation and Head Pose Generation via Diffusion Modelscited by 81
  5. 2024SMIRK: 3D Facial Expressions through Analysis-by-Neural-Synthesiscited by 81
  6. 2023FaceDiffuser: Speech-Driven 3D Facial Animation Synthesis Using Diffusioncited by 77
  7. 2023SelfTalk: A Self-Supervised Commutative Training Diagram to Comprehend 3D Talking Facescited by 67
  8. 2023ReMoS: 3D Motion-Conditioned Reaction Synthesis for Two-Person Interactionscited by 62
  9. 2024DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generationcited by 61
  10. 2023FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Modelscited by 55
  11. 2023NOFA: NeRF-based One-shot Facial Avatar Reconstructioncited by 53
  12. 2024CAP4D: Creating Animatable 4D Portrait Avatars with Morphable Multi-View Diffusion Modelscited by 52
38 more
  1. 2024VLOGGER: Multimodal Diffusion for Embodied Avatar Synthesiscited by 48
  2. 2024MambaTalk: Efficient Holistic Gesture Synthesis with Selective State Space Modelscited by 46
  3. 2024The Language of Motion: Unifying Verbal and Non-verbal Language of 3D Human Motioncited by 46
  4. 2023DPE: Disentanglement of Pose and Expression for General Video Portrait Editingcited by 45
  5. 2025GestureLSM: Latent Shortcut based Co-Speech Gesture Generation with Spatial-Temporal Modelingcited by 43
  6. 2024Media2Face: Co-speech Facial Animation Generation With Multi-Modality Guidancecited by 39
  7. 2024FlowVQTalker: High-Quality Emotional Talking Face Generation through Normalizing Flow and Quantizationcited by 39
  8. 2024Dyadic Interaction Modeling for Social Behavior Generationcited by 32
  9. 2024SemTalk: Holistic Co-speech Motion Generation with Frame-level Semantic Emphasiscited by 32
  10. 2023MedShapeNet -- A Large-Scale Dataset of 3D Medical Shapes for Computer Visioncited by 31
  11. 2025DualTalk: Dual-Speaker Interaction for 3D Talking Head Conversationscited by 31
  12. 2023FaceXHuBERT: Text-less Speech-driven E(X)pressive 3D Facial Animation Synthesis Using Self-Supervised Speech Representation Learningcited by 29
  13. 2024Towards Variable and Coordinated Holistic Co-Speech Motion Generationcited by 29
  14. 2024UniTalker: Scaling up Audio-Driven 3D Facial Animation through A Unified Modelcited by 29
  15. 2024MultiTalk: Enhancing 3D Talking Head Generation Across Languages with Multilingual Video Datasetcited by 28
  16. 2025ARTalk: Speech-Driven 3D Head Animation via Autoregressive Modelcited by 28
  17. 2023LaughTalk: Expressive 3D Talking Head Generation with Laughtercited by 27
  18. 2024Say Anything with Any Stylecited by 26
  19. 2024ScanTalk: 3D Talking Heads from Unregistered Scanscited by 22
  20. 20243D Face Tracking from 2D Video through Iterative Dense UV to Image Flowcited by 20
  21. 2024DiffSpeaker: Speech-Driven 3D Facial Animation with Diffusion Transformercited by 18
  22. 2023Speech-Driven 3D Face Animation with Composite and Regional Facial Movementscited by 17
  23. 2024DisCoRD: Discrete Tokens to Continuous Motion via Rectified Flow Decodingcited by 17
  24. 2025EchoMask: Speech-Queried Attention-based Mask Modeling for Holistic Co-Speech Motion Generationcited by 17
  25. 2023DiffusionTalker: Personalization and Acceleration for Speech-Driven 3D Face Diffusercited by 16
  26. 2023Mimic: Speaking Style Disentanglement for Speech-Driven 3D Facial Animationcited by 16
  27. 2024Learn2Talk: 3D Talking Face Learns from 2D Talking Facecited by 16
  28. 2024ReSyncer: Rewiring Style-based Generator for Unified Audio-Visually Synced Facial Performercited by 16
  29. 2024MMHead: Towards Fine-grained Multi-modal 3D Facial Animationcited by 16
  30. 2025MagicInfinite: Generating Infinite Talking Videos with Your Words and Voicecited by 16
  31. 2023Probabilistic Speech-Driven 3D Facial Motion Synthesis: New Benchmarks, Methods, and Applicationscited by 15
  32. 20233DiFACE: Diffusion-based Speech-driven 3D Facial Animation and Editingcited by 15
  33. 2024PersonaTalk: Bring Attention to Your Persona in Visual Dubbingcited by 15
  34. 2024KMTalk: Speech-Driven 3D Facial Animation with Key Motion Embeddingcited by 14
  35. 2023ExpCLIP: Bridging Text and Facial Expressions via Semantic Alignmentcited by 13
  36. 2024EmoVOCA: Speech-Driven Emotional 3D Talking Headscited by 13
  37. 2024MegActor-Sigma: Unlocking Flexible Mixed-Modal Control in Portrait Animation with Diffusion Transformercited by 13
  38. 2024ProbTalk3D: Non-Deterministic Emotion Controllable Speech-Driven 3D Facial Animation Synthesis Using VQ-VAEcited by 13

50 most cited of 183 All 183 (JSON)