mlsanityData to 7 Oct 2026

Face Animation with an Attribute-Guided Diffusion Model

arXiv 2304.03199 ↗First posted 6 Apr 2023cs.CVcited by 48

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Face animation has achieved much progress in computer vision. However, prevailing GAN-based methods suffer from unnatural distortions and artifacts due to sophisticated motion deformation. In this paper, we propose a Face Animation framework with an attribute-guided Diffusion Model (FADM), which is the first work to exploit the superior modeling capacity of diffusion models for photo-realistic talking-head generation. To mitigate the uncontrollable synthesis effect of the diffusion model, we design an Attribute-Guided Conditioning Network (AGCN) to adaptively combine the coarse animation features and 3D face reconstruction results, which can incorporate appearance and motion conditions into the diffusion process. These specific designs help FADM rectify unnatural artifacts and distortions, and also enrich high-fidelity facial details through iterative diffusion refinements with accurate animation attributes. FADM can flexibly and effectively improve existing animation videos. Extensive experiments on widely used talking-head benchmarks validate the effectiveness of FADM over prior arts.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 54 of its 64 references that are in the map. First posted on arXiv 6 Apr 2023.

Builds on

54 of 64 references.

Closest ancestors · 1

Same direction, newest first

  1. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesis21 Sep

Other references · 53

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  8. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  9. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  10. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  11. 2022Video Diffusion Modelscited by 2,280
  12. 2021Image Super-Resolution via Iterative Refinementcited by 1,506
38 more
  1. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,419
  2. 2021Palette: Image-to-Image Diffusion Modelscited by 1,296
  3. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  4. 2018VoxCeleb2: Deep Speaker Recognitioncited by 862
  5. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Modelcited by 768
  6. 2021Diffusion Probabilistic Models for 3D Point Cloud Generationcited by 734
  7. 2020Encoding in Style: a StyleGAN Encoder for Image-to-Image Translationcited by 727
  8. 2017VoxCeleb: a large-scale speaker identification datasetcited by 714
  9. 2020First Order Motion Model for Image Animationcited by 698
  10. 2017How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)cited by 524
  11. 20213D Shape Generation and Completion through Point-Voxel Diffusioncited by 516
  12. 2021Solving Inverse Problems in Medical Imaging with Score-Based Generative Modelscited by 514
  13. 2021Label-Efficient Semantic Segmentation with Diffusion Modelscited by 495
  14. 2020Learning an Animatable Detailed 3D Face Model from In-The-Wild Imagescited by 475
  15. 2020One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencingcited by 438
  16. 2021SRDiff: Single Image Super-Resolution with Diffusion Probabilistic Modelscited by 415
  17. 2021AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesiscited by 339
  18. 2020CurricularFace: Adaptive Curriculum Learning Loss for Deep Face Recognitioncited by 320
  19. 2021Score-based diffusion models for accelerated MRIcited by 308
  20. 2022Flexible Diffusion Modeling of Long Videoscited by 294
  21. 2019Few-Shot Adversarial Learning of Realistic Neural Talking Head Modelscited by 284
  22. 2020StyleRig: Rigging StyleGAN for 3D Control over Portrait Imagescited by 267
  23. 2018Deep Video Portraitscited by 265
  24. 2022Diffusion Probabilistic Modeling for Video Generationcited by 237
  25. 2023Implicit Diffusion Models for Continuous Super-Resolutioncited by 216
  26. 2018Animating Arbitrary Objects via Deep Motion Transfercited by 206
  27. 2021PIRenderer: Controllable Portrait Image Generation via Semantic Neural Renderingcited by 202
  28. 2019Few-shot Video-to-Video Synthesiscited by 196
  29. 2018X2Face: A network for controlling face generation by using images, audio, and pose codescited by 189
  30. 2022Depth-Aware Generative Adversarial Network for Talking Head Video Generationcited by 177
  31. 2022Latent Image Animator: Learning to Animate Images via Latent Space Navigationcited by 172
  32. 2020Fast Bi-layer Neural Synthesis of One-Shot Realistic Head Avatarscited by 116
  33. 2020Neural Head Reenactment with Latent Pose Descriptorscited by 106
  34. 2020HeadGAN: One-shot Neural Head Synthesis and Editingcited by 105
  35. 2021A Conditional Point Diffusion-Refinement Paradigm for 3D Point Cloud Completioncited by 100
  36. 2019MarioNETte: Few-shot Face Reenactment Preserving Identity of Unseen Targetscited by 90
  37. 2020GIF: Generative Interpretable Facescited by 53
  38. 2019Neural Style-Preserving Visual Dubbingcited by 47

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 17; 2025: 21; 2026: 7.

48 papers.

Sort:

By area:

  1. 2024LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Controlcited by 202
  2. 2024Follow-Your-Emoji: Fine-Controllable and Expressive Freestyle Portrait Animationcited by 172
  3. 2024X-Portrait: Expressive Portrait Animation with Hierarchical Motion Attentioncited by 88
  4. 2024AniTalker: Animate Vivid and Diverse Talking Faces through Identity-Decoupled Facial Motion Encodingcited by 67
  5. 2024Face Adapter for Pre-Trained Diffusion Models with Fine-Grained ID and Attribute Controlcited by 66
  6. 2025Any2AnyTryon: Leveraging Adaptive Position Embeddings for Versatile Virtual Clothing Taskscited by 62
  7. 2023DiffusionAvatars: Deferred Diffusion for High-fidelity 3D Head Avatarscited by 42
  8. 2025Follow-Your-Emoji-Faster: Towards Efficient, Fine-Controllable, and Expressive Freestyle Portrait Animationcited by 41
  9. 2024EditWorld: Simulating World Dynamics for Instruction-Following Image Editingcited by 37
  10. 2023Controllable Mind Visual Diffusion Modelcited by 29
  11. 2024Trans4D: Realistic Geometry-Aware Transition for Compositional Text-to-4D Synthesiscited by 19
  12. 2023A Generalist FaceX via Learning Unified Facial Representationcited by 13
36 more
  1. 2025LVFace: Progressive Cluster Optimization for Large Vision Models in Face Recognitioncited by 12
  2. 2025FaceShot: Bring Any Character into Lifecited by 12
  3. 2025FantasyPortrait: Enhancing Multi-Character Portrait Animation with Expression-Augmented Diffusion Transformerscited by 12
  4. 2024CemiFace: Center-based Semi-hard Synthetic Face Generation for Face Recognitioncited by 10
  5. 2025Monocular and Generalizable Gaussian Talking Head Animationcited by 10
  6. 2024DiffuX2CT: Diffusion Learning to Reconstruct CT Images from Biplanar X-Rayscited by 9
  7. 2024DH-FaceVid-1K: A Large-Scale High-Quality Dataset for Face Video Generationcited by 8
  8. 2024Synergizing Motion and Appearance: Multi-Scale Compensatory Codebooks for Talking Head Video Generationcited by 8
  9. 2025High-Fidelity Relightable Monocular Portrait Animation with Lighting-Controllable Video Diffusion Modelcited by 8
  10. 2025PersonaLive! Expressive Portrait Image Animation for Live Streamingcited by 6
  11. 2024Takin-ADA: Emotion Controllable Audio-Driven Animation with Canonical and Landmark Loss Optimizationcited by 4
  12. 2024EmojiDiff: Advanced Facial Expression Control with High Identity Preservation in Portrait Generationcited by 4
  13. 2025InterAnimate: Taming Region-aware Diffusion Model for Realistic Human Interaction Animationcited by 4
  14. 2025Towards Consistent and Controllable Image Synthesis for Face Editingcited by 3
  15. 2025Controlling Avatar Diffusion with Learnable Gaussian Embeddingcited by 3
  16. 2025MagicPortrait: Temporally Consistent Face Reenactment with 3D Geometric Guidancecited by 3
  17. 2025In-Context Sync-LoRA for Portrait Video Editingcited by 3
  18. 2026OpenWorldLib: A Unified Codebase and Definition of Advanced World Modelscited by 3
  19. 2024MIMAFace: Face Animation via Motion-Identity Modulated Appearance Feature Learningcited by 2
  20. 2025Unlock Pose Diversity: Accurate and Efficient Implicit Keypoint-based Spatiotemporal Diffusion for Audio-driven Talking Portraitcited by 2
  21. 2025High-Quality Facial Albedo Generation for 3D Face Reconstruction from a Single Image using a Coarse-to-Fine Approachcited by 2
  22. 2026Now You See Me, Now You Don't: A Unified Framework for Expression Consistent Anonymization in Talking Head Videoscited by 2
  23. 2026Deepfake Synthesis vs. Detection: An Uneven Contestcited by 2
  24. 2024AniFaceDiff: Animating Stylized Avatars via Parametric Conditioned Diffusion Modelscited by 1
  25. 2025Test-Time Augmentation for Pose-invariant Face Recognitioncited by 1
  26. 2025FactorPortrait: Controllable Portrait Animation via Disentangled Expression, Pose, and Viewpointcited by 1
  27. 2024Video-Driven Animation of Neural Head Avatarscited by 0
  28. 2024CFCPalsy: Facial Image Synthesis with Cross-Fusion Cycle Diffusion Model for Facial Paralysis Individualscited by 0
  29. 20253D Engine-ready Photorealistic Avatars via Dynamic Texturescited by 0
  30. 2025HairShifter: Consistent and High-Fidelity Video Hair Transfer via Anchor-Guided Animationcited by 0
  31. 2025MagicAnime: A Hierarchically Annotated, Multimodal and Multitasking Dataset with Benchmarks for Cartoon Animation Generationcited by 0
  32. 2025JOLT3D: Joint Learning of Talking Heads and 3DMM Parameters with Application to Lip-Synccited by 0
  33. 2026Motion Manipulation via Unsupervised Keypoint Positioning in Face Animationcited by 0
  34. 2026GSwap: Realistic Head Swapping with Dynamic Neural Gaussian Fieldcited by 0
  35. 2026FG-Portrait: 3D Flow Guided Editable Portrait Animationcited by 0
  36. 2026fMRI2Face: A Full-HD fMRI-Video Dataset and Geometry-Guided Neural Decoding Framework for Dynamic Human Face Reconstructioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023DiffusionTalker: Personalization and Acceleration for Speech-Driven 3D Face Diffuser
  2. 2023Diffused Heads: Diffusion Models Beat GANs on Talking-Face Generation
  3. 20233DiFACE: Diffusion-based Speech-driven 3D Facial Animation and Editing
  4. 2024AnimateMe: 4D Facial Expressions via Diffusion Models
  5. 2024IF-MDM: Implicit Face Motion Diffusion Model for High-Fidelity Realtime Talking Head Generation
  6. 2025MoDiT: Learning Highly Consistent 3D Motion Coefficients with Diffusion Transformer for Talking Head Generation
  7. 2024DiffSpeaker: Speech-Driven 3D Facial Animation with Diffusion Transformer
  8. 2025Stable Video-Driven Portraits