mlsanityData to 14 Oct 2026

MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglement

arXiv 2104.08223 ↗First posted 16 Apr 2021cs.CVcited by 190

Not on the map. Why?

Cited by 20 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This paper presents a generic method for generating full facial 3D animation from speech. Existing approaches to audio-driven facial animation exhibit uncanny or static upper face animation, fail to produce accurate and plausible co-articulation or rely on person-specific models that limit their scalability. To improve upon existing models, we propose a generic audio-driven facial animation approach that achieves highly realistic motion synthesis results for the entire face. At the core of our approach is a categorical latent space for facial animation that disentangles audio-correlated and audio-uncorrelated information based on a novel cross-modality loss. Our approach ensures highly accurate lip motion, while also synthesizing plausible animation of the parts of the face that are uncorrelated to the audio signal, such as eye blinks and eye brow motion. We demonstrate that our approach outperforms several baselines and obtains state-of-the-art quality both qualitatively and quantitatively. A perceptual user study demonstrates that our approach is deemed more realistic than the current state-of-the-art in over 75% of cases. We recommend watching the supplemental video before reading the paper: https://github.com/facebookresearch/meshtalk

Key facts. Cited by 190 papers in the map (data through 14 Oct 2026). Builds on 20 of its 42 references that are in the map. First posted on arXiv 16 Apr 2021.

Builds on

20 of 42 references.

References · 20

  1. 2017Attention Is All You Needcited by 60,361
  2. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  3. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  4. 2016Conditional Image Generation with PixelCNN Decoderscited by 862
  5. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 672
  6. 2014Deep Speech: Scaling up end-to-end speech recognitioncited by 354
  7. 2019What Makes Training Multi-Modal Classification Networks Hard?cited by 344
  8. 2019Capture, Learning, and Synthesis of 3D Speaking Stylescited by 259
  9. 2019Neural Voice Puppetry: Audio-driven Facial Reenactmentcited by 250
  10. 2018Talking Face Generation by Adversarially Disentangled Audio-Visual Representationcited by 212
  11. 2018X2Face: A network for controlling face generation by using images, audio, and pose codescited by 189
  12. 2019Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Modelscited by 178
8 more
  1. 2018Deep Appearance Models for Face Renderingcited by 164
  2. 2017You said that?cited by 99
  3. 2018Talking Face Generation by Conditional Recurrent Adversarial Networkcited by 84
  4. 2018VisemeNet: Audio-Driven Animator-Centric Speech Animationcited by 63
  5. 2020Audio- and Gaze-driven Facial Animation of Codec Avatarscited by 46
  6. 2018End-to-End Speech-Driven Facial Animation with Temporal GANscited by 31
  7. 2020Expressive Telepresence via Modular Codec Avatarscited by 17
  8. 2018Generating Talking Face Landmarks from Speechcited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 4; 2022: 13; 2023: 51; 2024: 44; 2025: 43; 2026: 35.

190 papers.

Sort:

By area:

  1. 2022TEMOS: Generating diverse human motions from textual descriptionscited by 492
  2. 2021FaceFormer: Speech-Driven 3D Facial Animation with Transformerscited by 209
  3. 2023Multimodal Large Language Models: A Surveycited by 195
  4. 2023CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Priorcited by 184
  5. 2022Generating Holistic 3D Human Motion from Speechcited by 180
  6. 2022BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesiscited by 178
  7. 2023EmoTalk: Speech-Driven Emotional Disentanglement for 3D Face Animationcited by 142
  8. 2023EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modelingcited by 105
  9. 2023Emotional Speech-Driven Animation with Content-Emotion Disentanglementcited by 98
  10. 2023StyleSync: High-Fidelity Generalized and Personalized Lip Sync in Style-based Generatorcited by 89
  11. 2022Multiface: A Dataset for Neural Face Renderingcited by 87
  12. 2023A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need?cited by 84
38 more
  1. 2024SMIRK: 3D Facial Expressions through Analysis-by-Neural-Synthesiscited by 84
  2. 2024From Audio to Photoreal Embodiment: Synthesizing Humans in Conversationscited by 82
  3. 2023DiffPoseTalk: Speech-Driven Stylistic 3D Facial Animation and Head Pose Generation via Diffusion Modelscited by 81
  4. 2023FaceDiffuser: Speech-Driven 3D Facial Animation Synthesis Using Diffusioncited by 78
  5. 2022Imitator: Personalized Speech-driven 3D Facial Animationcited by 74
  6. 2023SelfTalk: A Self-Supervised Commutative Training Diagram to Comprehend 3D Talking Facescited by 68
  7. 2021Responsive Listening Head Generation: A Benchmark Dataset and Baselinecited by 67
  8. 2024DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generationcited by 62
  9. 2022FLAG: Flow-based 3D Avatar Generation from Sparse Observationscited by 55
  10. 2022Audio-Driven Co-Speech Gesture Video Generationcited by 55
  11. 2023FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Modelscited by 55
  12. 2024VLOGGER: Multimodal Diffusion for Embodied Avatar Synthesiscited by 50
  13. 2023RenderMe-360: A Large Digital Asset Library and Benchmarks Towards High-fidelity Head Avatarscited by 45
  14. 2021Neural Dubber: Dubbing for Videos According to Scriptscited by 39
  15. 2022u-HuBERT: Unified Mixed-Modal Speech Pretraining And Zero-Shot Transfer to Unlabeled Modalitycited by 39
  16. 2024Media2Face: Co-speech Facial Animation Generation With Multi-Modality Guidancecited by 39
  17. 2022Rhythmic Gesticulator: Rhythm-Aware Co-Speech Gesture Synthesis with Hierarchical Neural Embeddingscited by 38
  18. 2024Dyadic Interaction Modeling for Social Behavior Generationcited by 32
  19. 2025DualTalk: Dual-Speaker Interaction for 3D Talking Head Conversationscited by 32
  20. 2022DialogueNeRF: Towards Realistic Avatar Face-to-Face Conversation Video Generationcited by 30
  21. 2024UniTalker: Scaling up Audio-Driven 3D Facial Animation through A Unified Modelcited by 30
  22. 2023FaceXHuBERT: Text-less Speech-driven E(X)pressive 3D Facial Animation Synthesis Using Self-Supervised Speech Representation Learningcited by 29
  23. 2025ARTalk: Speech-Driven 3D Head Animation via Autoregressive Modelcited by 29
  24. 2023Speech Driven Video Editing via an Audio-Conditioned Diffusion Modelcited by 28
  25. 2024MultiTalk: Enhancing 3D Talking Head Generation Across Languages with Multilingual Video Datasetcited by 28
  26. 2023LaughTalk: Expressive 3D Talking Head Generation with Laughtercited by 27
  27. 2022Audio-Visual Speech Codecs: Rethinking Audio-Visual Speech Enhancement by Re-Synthesiscited by 24
  28. 2024ScanTalk: 3D Talking Heads from Unregistered Scanscited by 24
  29. 2024GaussianTalker: Speaker-specific Talking Head Synthesis via 3D Gaussian Splattingcited by 24
  30. 2024CustomListener: Text-guided Responsive Interaction for User-friendly Listening Head Generationcited by 22
  31. 2022Deep Learning for Visual Speech Analysis: A Surveycited by 21
  32. 2024DiffSpeaker: Speech-Driven 3D Facial Animation with Diffusion Transformercited by 18
  33. 2023Speech-Driven 3D Face Animation with Composite and Regional Facial Movementscited by 17
  34. 2023Emotional Listener Portrait: Neural Listener Head Generation with Emotioncited by 17
  35. 2023DiffusionTalker: Personalization and Acceleration for Speech-Driven 3D Face Diffusercited by 16
  36. 2023Mimic: Speaking Style Disentanglement for Speech-Driven 3D Facial Animationcited by 16
  37. 2024Learn2Talk: 3D Talking Face Learns from 2D Talking Facecited by 16
  38. 2024EmoTalk3D: High-Fidelity Free-View Synthesis of Emotional 3D Talking Headcited by 16

50 most cited of 190 All 190 (JSON)