mlsanityData to 7 Oct 2026

3DiFACE: Diffusion-based Speech-driven 3D Facial Animation and Editing

arXiv 2312.00870 ↗First posted 1 Dec 2023cs.CV cs.AI cs.GRcited by 15

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

We present 3DiFACE, a novel method for personalized speech-driven 3D facial animation and editing. While existing methods deterministically predict facial animations from speech, they overlook the inherent one-to-many relationship between speech and facial expressions, i.e., there are multiple reasonable facial expression animations matching an audio input. It is especially important in content creation to be able to modify generated motion or to specify keyframes. To enable stochasticity as well as motion editing, we propose a lightweight audio-conditioned diffusion model for 3D facial motion. This diffusion model can be trained on a small 3D motion dataset, maintaining expressive lip motion output. In addition, it can be finetuned for specific subjects, requiring only a short video of the person. Through quantitative and qualitative evaluations, we show that our method outperforms existing state-of-the-art techniques and yields speech-driven animations with greater fidelity and diversity.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 57 of its 75 references that are in the map. First posted on arXiv 1 Dec 2023.

Builds on

57 of 75 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023DiffPoseTalk: Speech-Driven Stylistic 3D Facial Animation and Head Pose Generation via Diffusion Models30 Sep
  2. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesis21 Sep

Other references · 55

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  7. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  8. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  9. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  10. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  11. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  12. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
38 more
  1. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  2. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  3. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
  4. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  5. 2019FaceForensics++: Learning to Detect Manipulated Facial Imagescited by 1,095
  6. 2022Human Motion Diffusion Modelcited by 1,062
  7. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Modelcited by 768
  8. 2023Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creationcited by 684
  9. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 620
  10. 2020Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstructioncited by 465
  11. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 463
  12. 20183D human pose estimation in video with temporal convolutions and semi-supervised trainingcited by 401
  13. 2022PhysDiff: Physics-Guided Human Motion Diffusion Modelcited by 365
  14. 2014Deep Speech: Scaling up end-to-end speech recognitioncited by 354
  15. 2021AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesiscited by 339
  16. 2020MakeItTalk: Speaker-Aware Talking-Head Animationcited by 310
  17. 20193D Morphable Face Models -- Past, Present and Futurecited by 269
  18. 2019Capture, Learning, and Synthesis of 3D Speaking Stylescited by 257
  19. 2022EMOCA: Emotion Driven Monocular Face Capture and Animationcited by 254
  20. 2019Neural Voice Puppetry: Audio-driven Facial Reenactmentcited by 249
  21. 2021Neural Head Avatars from Monocular RGB Videoscited by 212
  22. 2021FaceFormer: Speech-Driven 3D Facial Animation with Transformerscited by 208
  23. 2021MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglementcited by 189
  24. 2023CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Priorcited by 183
  25. 2022Towards Metrical Reconstruction of Human Facescited by 165
  26. 2022Instant Volumetric Head Avatarscited by 157
  27. 2021Audio2Head: Audio-driven One-shot Talking-head Generation with Natural Head Motioncited by 148
  28. 2019Realistic Speech-Driven Facial Animation with GANscited by 142
  29. 2023EmoTalk: Speech-Driven Emotional Disentanglement for 3D Face Animationcited by 141
  30. 2023Text-to-3D with Classifier Score Distillationcited by 127
  31. 2023State of the Art on Diffusion Models for Visual Computingcited by 126
  32. 2017You said that?cited by 99
  33. 2023Emotional Speech-Driven Animation with Content-Emotion Disentanglementcited by 98
  34. 2020Audio-driven Talking Face Video Generation with Learning-based Personalized Head Posecited by 92
  35. 2020Everybody's Talkin': Let Me Talk as You Wantcited by 83
  36. 2022DFA-NeRF: Personalized Talking Head Generation via Disentangled Face Attributes Neural Renderingcited by 80
  37. 2023FaceDiffuser: Speech-Driven 3D Facial Animation Synthesis Using Diffusioncited by 77
  38. 2022Imitator: Personalized Speech-driven 3D Facial Animationcited by 73

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 3; 2026: 3.

15 papers.

Sort:

By area:

  1. 2024Media2Face: Co-speech Facial Animation Generation With Multi-Modality Guidancecited by 39
  2. 2024Gaussian Eigen Models for Human Headscited by 21
  3. 2024DiffSpeaker: Speech-Driven 3D Facial Animation with Diffusion Transformercited by 18
  4. 2024MMHead: Towards Fine-grained Multi-modal 3D Facial Animationcited by 16
  5. 2024ProbTalk3D: Non-Deterministic Emotion Controllable Speech-Driven 3D Facial Animation Synthesis Using VQ-VAEcited by 13
  6. 2025Wav2Sem: Plug-and-Play Audio Semantic Decoupling for 3D Speech-Driven Facial Animationcited by 8
  7. 2024AnimateMe: 4D Facial Expressions via Diffusion Modelscited by 6
  8. 2024Combo: Co-speech holistic 3D human motion generation and efficient customizable adaptation in harmonycited by 6
  9. 2025Advancing Talking Head Generation: A Comprehensive Survey of Multi-Modal Methodologies, Datasets, Evaluation Metrics, and Loss Functionscited by 6
  10. 2024Diverse Code Query Learning for Speech-Driven Facial Animationcited by 2
  11. 20243DFacePolicy: Audio-Driven 3D Facial Animation Based on Action Controlcited by 1
  12. 2025EmoDiffusion: Enhancing Emotional 3D Facial Animation with Latent Diffusion Modelscited by 0
3 more
  1. 2026ExpressEdit: Fast Editing of Stylized Facial Expressions with Diffusion Models in Photoshopcited by 0
  2. 2026Polyglot: Multilingual Style Preserving Speech-Driven Facial Animationcited by 0
  3. 2026AnyTalk: Speech Animation for Arbitrary Characters Leveraging a Video Generation Modelcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 20253DiFACE: Synthesizing and Editing Holistic 3D Facial Animation
  2. 2023SAiD: Speech-driven Blendshape Facial Animation with Diffusion
  3. 2025StreamingTalker: Audio-driven 3D Facial Animation with Autoregressive Diffusion Model
  4. 2023DiffusionTalker: Personalization and Acceleration for Speech-Driven 3D Face Diffuser
  5. 2025Model See Model Do: Speech-Driven Facial Animation with Style Control
  6. 2025DiffusionTalker: Efficient and Compact Speech-Driven 3D Talking Head via Personalizer-Guided Distillation
  7. 2023Speech Driven Video Editing via an Audio-Conditioned Diffusion Model
  8. 2025Controllable Expressive 3D Facial Animation via Diffusion in a Unified Multimodal Space