mlsanityData to 7 Oct 2026

DiffusionAct: Controllable Diffusion Autoencoder for One-shot Face Reenactment

arXiv 2403.17217 ↗First posted 25 Mar 2024cs.CV cs.AIcited by 14

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Video-driven neural face reenactment aims to synthesize realistic facial images that successfully preserve the identity and appearance of a source face, while transferring the target head pose and facial expressions. Existing GAN-based methods suffer from either distortions and visual artifacts or poor reconstruction quality, i.e., the background and several important appearance details, such as hair style/color, glasses and accessories, are not faithfully reconstructed. Recent advances in Diffusion Probabilistic Models (DPMs) enable the generation of high-quality realistic images. To this end, in this paper we present DiffusionAct, a novel method that leverages the photo-realistic image generation of diffusion models to perform neural face reenactment. Specifically, we propose to control the semantic space of a Diffusion Autoencoder (DiffAE), in order to edit the facial pose of the input images, defined as the head pose orientation and the facial expressions. Our method allows one-shot, self, and cross-subject reenactment, without requiring subject-specific fine-tuning. We compare against state-of-the-art GAN-, StyleGAN2-, and diffusion-based methods, showing better or on-par reenactment performance.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 65 of its 78 references that are in the map. First posted on arXiv 25 Mar 2024.

Builds on

65 of 78 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024InstantID: Zero-shot Identity-Preserving Generation in Seconds15 Jan
  2. 2023Face Aging via Diffusion-based Editing20 Sep
  3. 2023Effective Real Image Editing with Accelerated Iterative Diffusion Inversion10 Sep
  4. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models13 Aug
  5. 2023ChatFace: Chat-Guided Real Face Editing via Diffusion Latent Space Manipulation24 May
  6. 2023Collaborative Diffusion for Multi-Modal Face Generation and Editing20 Apr
  7. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  8. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Models17 Nov
  9. 2021DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation6 Oct

Other references · 56

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  6. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  7. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  8. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  9. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,678
  10. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  11. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,419
  12. 2016HyperNetworkscited by 1,003
38 more
  1. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  2. 2018VoxCeleb2: Deep Speaker Recognitioncited by 862
  3. 2017VoxCeleb: a large-scale speaker identification datasetcited by 714
  4. 2020First Order Motion Model for Image Animationcited by 698
  5. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 669
  6. 2020GANSpace: Discovering Interpretable GAN Controlscited by 612
  7. 2021Designing an Encoder for StyleGAN Image Manipulationcited by 542
  8. 2017How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)cited by 524
  9. 2020Learning an Animatable Detailed 3D Face Model from In-The-Wild Imagescited by 475
  10. 2021Diffusion Autoencoders: Toward a Meaningful and Decodable Representationcited by 447
  11. 2020One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencingcited by 438
  12. 2021Pivotal Tuning for Latent-based Editing of Real Imagescited by 427
  13. 2020BiSeNet V2: Bilateral Network with Guided Aggregation for Real-time Semantic Segmentationcited by 294
  14. 2020Unsupervised Discovery of Interpretable Directions in the GAN Latent Spacecited by 288
  15. 2019Few-Shot Adversarial Learning of Realistic Neural Talking Head Modelscited by 284
  16. 2022EMOCA: Emotion Driven Monocular Face Capture and Animationcited by 254
  17. 2022Thin-Plate Spline Motion Model for Image Animationcited by 209
  18. 2021PIRenderer: Controllable Portrait Image Generation via Semantic Neural Renderingcited by 202
  19. 2021General Facial Representation Learning in a Visual-Linguistic Mannercited by 193
  20. 2022Depth-Aware Generative Adversarial Network for Talking Head Video Generationcited by 177
  21. 2022VFHQ: A High-Quality Dataset and Benchmark for Video Face Super-Resolutioncited by 174
  22. 2022StyleHEAT: One-Shot High-Resolution Editable Talking Face Generation via Pre-trained StyleGANcited by 169
  23. 2023Diffused Heads: Diffusion Models Beat GANs on Talking-Face Generationcited by 142
  24. 2020ETH-XGaze: A Large Scale Dataset for Gaze Estimation under Extreme Head Pose and Gaze Variationcited by 141
  25. 2023DiffTalk: Crafting Diffusion Models for Generalized Audio-Driven Portraits Animationcited by 136
  26. 2022Realistic One-shot Mesh-based Head Avatarscited by 119
  27. 2020Fast Bi-layer Neural Synthesis of One-Shot Realistic Head Avatarscited by 116
  28. 2020Neural Head Reenactment with Latent Pose Descriptorscited by 106
  29. 2023StyleSync: High-Fidelity Generalized and Personalized Lip Sync in Style-based Generatorcited by 89
  30. 2023Efficient Emotional Adaptation for Audio-Driven Talking-Head Generationcited by 81
  31. 2023DiffusionRig: Learning Personalized Priors for Facial Appearance Editingcited by 74
  32. 2022Stitch it in Time: GAN-Based Facial Editing of Real Videoscited by 68
  33. 2021WarpedGANSpace: Finding non-linear RBF paths in GAN latent spacecited by 48
  34. 2023DAE-Talker: High Fidelity Speech-Driven Talking Face Generation with Diffusion Autoencodercited by 48
  35. 2023HyperReenact: One-Shot Reenactment via Jointly Learning to Refine and Retarget Facescited by 48
  36. 2023DPE: Disentanglement of Pose and Expression for General Video Portrait Editingcited by 45
  37. 2023Attribute-preserving Face Dataset Anonymization via Latent Code Optimizationcited by 43
  38. 2022Diffusion Video Autoencoders: Toward Temporally Consistent Face Video Editing via Disentangled Video Encodingcited by 39

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 3; 2026: 4.

14 papers.

Sort:

By area:

  1. 2024Deepfake Generation and Detection: A Benchmark and Surveycited by 66
  2. 2024VOODOO XP: Expressive One-Shot Head Reenactment for VR Telepresencecited by 19
  3. 2024Joker: Conditional 3D Head Synthesis with Extreme Facial Expressionscited by 17
  4. 2024Are CLIP features all you need for Universal Synthetic Image Origin Attribution?cited by 14
  5. 2024Deepfake Media Generation and Detection in the Generative AI Era: A Survey and Outlookcited by 10
  6. 2024MM2Latent: Text-to-facial image generation and editing in GANs with multimodal assistancecited by 7
  7. 2024Anchored Diffusion for Video Face Reenactmentcited by 2
  8. 2025Revisiting Diffusion Autoencoder Training for Image Reconstruction Qualitycited by 0
  9. 2025FaceEditTalker: Controllable Talking Head Generation with Facial Attribute Editingcited by 0
  10. 2025Navigating Large-Pose Challenge for High-Fidelity Face Reenactment with Video Diffusion Modelcited by 0
  11. 2026Diff-PC: Identity-preserving and 3D-aware Controllable Diffusion for Zero-shot Portrait Customizationcited by 0
  12. 2026GazeCLIP: Gaze-Guided CLIP with Adaptive-Enhanced Fine-Grained Language Prompt for Deepfake Attribution and Detectioncited by 0
2 more
  1. 2026On Applicability of Synthetic Datasets for Facial Expression Recognitioncited by 0
  2. 2026Multi-Tool Image Editing Attribution in Facial Forgerycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Face Adapter for Pre-Trained Diffusion Models with Fine-Grained ID and Attribute Control
  2. 2023FaceDNeRF: Semantics-Driven Face Reconstruction, Prompt Editing and Relighting with Diffusion Models
  3. 2024Diffusion-driven GAN Inversion for Multi-Modal Face Image Generation
  4. 2024Controllable Face Synthesis with Semantic Latent Diffusion Models
  5. 2022DiffFace: Diffusion-based Face Swapping with Facial Guidance
  6. 2024OSDFace: One-Step Diffusion Model for Face Restoration
  7. 2023Conditioning Diffusion Models via Attributes and Semantic Masks for Face Generation
  8. 2024Realistic and Efficient Face Swapping: A Unified Approach with Diffusion Models