# Realistic and Efficient Face Swapping: A Unified Approach with Diffusion Models

arXiv: https://arxiv.org/abs/2409.07269 · HTML: https://mlsanity.com/paper/2409.07269/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 28 of its 37 references that are in the map.
- First posted on arXiv 11 Sep 2024.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 28 of 28)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [FaceForensics++: Learning to Detect Manipulated Facial Images](https://mlsanity.com/paper/1901.08971/) (2019)
- [Pseudo Numerical Methods for Diffusion Models on Manifolds](https://mlsanity.com/paper/2202.09778/) (2022)
- [MaskGAN: Towards Diverse and Interactive Facial Image Manipulation](https://mlsanity.com/paper/1907.11922/) (2019)
- [Paint by Example: Exemplar-based Image Editing with Diffusion Models](https://mlsanity.com/paper/2211.13227/) (2022)
- [Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Set](https://mlsanity.com/paper/1903.08527/) (2019)
- [FSGAN: Subject Agnostic Face Swapping and Reenactment](https://mlsanity.com/paper/1908.05932/) (2019)
- [SimSwap: An Efficient Framework For High Fidelity Face Swapping](https://mlsanity.com/paper/2106.06340/) (2021)
- [FaceShifter: Towards High Fidelity And Occlusion Aware Face Swapping](https://mlsanity.com/paper/1912.13457/) (2019)
- [HOPE-Net: A Graph-based Model for Hand-Object Pose Estimation](https://mlsanity.com/paper/2004.00060/) (2020)
- [One Shot Face Swapping on Megapixels](https://mlsanity.com/paper/2105.04932/) (2021)
- [HifiFace: 3D Shape and Semantic Prior Guided High Fidelity Face Swapping](https://mlsanity.com/paper/2106.09965/) (2021)
- [DiffFace: Diffusion-based Face Swapping with Facial Guidance](https://mlsanity.com/paper/2212.13344/) (2022)
- [Towards Open-Set Identity Preserving Face Synthesis](https://mlsanity.com/paper/1803.11182/) (2018)
- [Fine-Grained Face Swapping via Regional GAN Inversion](https://mlsanity.com/paper/2211.14068/) (2022)
- [High-resolution Face Swapping via Latent Semantics Disentanglement](https://mlsanity.com/paper/2203.15958/) (2022)
- [Region-Aware Face Swapping](https://mlsanity.com/paper/2203.04564/) (2022)
- [FaceDancer: Pose- and Occlusion-Aware High Fidelity Face Swapping](https://mlsanity.com/paper/2210.10473/) (2022)
- [Few-Shot Head Swapping in the Wild](https://mlsanity.com/paper/2204.13100/) (2022)
- [Boosting Adversarial Transferability across Model Genus by Deformation-Constrained Warping](https://mlsanity.com/paper/2402.03951/) (2024)
- [HS-Diffusion: Semantic-Mixing Diffusion for Head Swapping](https://mlsanity.com/paper/2212.06458/) (2022)
- [FReeNet: Multi-Identity Face Reenactment](https://mlsanity.com/paper/1905.11805/) (2019)

## Built on by (most cited first, 30 of 36)

- [Deepfake Detection that Generalizes Across Benchmarks](https://mlsanity.com/paper/2508.06248/) (2025)
- [CanonSwap: High-Fidelity and Consistent Video Face Swapping via Canonical Space Modulation](https://mlsanity.com/paper/2507.02691/) (2025)
- [LoRAShop: Training-Free Multi-Concept Image Generation and Editing with Rectified Flow Transformers](https://mlsanity.com/paper/2505.23758/) (2025)
- [DreamID: High-Fidelity and Fast diffusion-based Face Swapping via Triplet ID Group Learning](https://mlsanity.com/paper/2504.14509/) (2025)
- [DreamID-V:Bridging the Image-to-Video Gap for High-Fidelity Face Swapping via Diffusion Transformer](https://mlsanity.com/paper/2601.01425/) (2026)
- [VividFace: A Diffusion-Based Hybrid Framework for High-Fidelity Video Face Swapping](https://mlsanity.com/paper/2412.11279/) (2024)
- [GHOST 2.0: generative high-fidelity one shot transfer of heads](https://mlsanity.com/paper/2502.18417/) (2025)
- [High-Fidelity Diffusion Face Swapping with ID-Constrained Facial Conditioning](https://mlsanity.com/paper/2503.22179/) (2025)
- [Benchmarking Unified Face Attack Detection via Hierarchical Prompt Tuning](https://mlsanity.com/paper/2505.13327/) (2025)
- [My Face Is Mine, Not Yours: Facial Protection Against Diffusion Model Face Swapping](https://mlsanity.com/paper/2505.15336/) (2025)
- [Zero-Shot Head Swapping in Real-World Scenarios](https://mlsanity.com/paper/2503.00861/) (2025)
- [UniAIDet: A Unified and Universal Benchmark for AI-Generated Image Content Detection and Localization](https://mlsanity.com/paper/2510.23023/) (2025)
- [PVLM: Parsing-Aware Vision Language Model with Dynamic Contrastive Learning for Zero-Shot Deepfake Attribution](https://mlsanity.com/paper/2504.14129/) (2025)
- [De-Fake: Style based Anomaly Deepfake Detection](https://mlsanity.com/paper/2507.03334/) (2025)
- [Towards Robust Protective Perturbation against DeepFake Face Swapping](https://mlsanity.com/paper/2512.07228/) (2025)
- [AlphaFace: High Fidelity and Real-time Face Swapper Robust to Facial Pose](https://mlsanity.com/paper/2601.16429/) (2026)
- [Beauty and the Beast: Imperceptible Perturbations Against Diffusion-Based Face Swapping via Directional Attribute Editing](https://mlsanity.com/paper/2601.22744/) (2026)
- [Towards High Fidelity Face Swapping: A Comprehensive Survey and New Benchmark](https://mlsanity.com/paper/2605.00883/) (2026)
- [Assessing the Use of Face Swapping Methods as Face Anonymizers in Videos](https://mlsanity.com/paper/2505.20985/) (2025)
- [A Controllable 3D Deepfake Generation Framework with Gaussian Splatting](https://mlsanity.com/paper/2509.11624/) (2025)
- [DiffSwap++: 3D Latent-Controlled Diffusion for Identity-Preserving Face Swapping](https://mlsanity.com/paper/2511.05575/) (2025)
- [Layer-Aware Video Composition via Split-then-Merge](https://mlsanity.com/paper/2511.20809/) (2025)
- [DirectSwap: Paired, Mask-Free Video Head Swapping with Full-Reference Evaluation](https://mlsanity.com/paper/2512.09417/) (2025)
- [GaussianSwap: Animatable Video Face Swapping with 3D Gaussian Splatting](https://mlsanity.com/paper/2601.05511/) (2026)
- [S3POT: Contrast-Driven Face Occlusion Segmentation via Self-Supervised Prompt Learning](https://mlsanity.com/paper/2602.00635/) (2026)
- [VFace: A Training-Free Approach for Diffusion-Based Video Face Swapping](https://mlsanity.com/paper/2602.07835/) (2026)
- [Optimizing ID Consistency in Multimodal Large Models: Facial Restoration via Alignment, Entanglement, and Disentanglement](https://mlsanity.com/paper/2602.18752/) (2026)
- [Beyond Detection: Multi-Scale Hidden-Code for Natural Image Deepfake Recovery and Factual Retrieval](https://mlsanity.com/paper/2602.22759/) (2026)
- [UniBioTransfer: A Unified Framework for Multiple Biometrics Transfer](https://mlsanity.com/paper/2603.19637/) (2026)
- [GSwap: Realistic Head Swapping with Dynamic Neural Gaussian Field](https://mlsanity.com/paper/2603.23168/) (2026)

## Abstract (arXiv)

Despite promising progress in face swapping task, realistic swapped images remain elusive, often marred by artifacts, particularly in scenarios involving high pose variation, color differences, and occlusion. To address these issues, we propose a novel approach that better harnesses diffusion models for face-swapping by making following core contributions. (a) We propose to re-frame the face-swapping task as a self-supervised, train-time inpainting problem, enhancing the identity transfer while blending with the target image. (b) We introduce a multi-step Denoising Diffusion Implicit Model (DDIM) sampling during training, reinforcing identity and perceptual similarities. (c) Third, we introduce CLIP feature disentanglement to extract pose, expression, and lighting information from the target image, improving fidelity. (d) Further, we introduce a mask shuffling technique during inpainting training, which allows us to create a so-called universal model for swapping, with an additional feature of head swapping. Ours can swap hair and even accessories, beyond traditional face swapping. Unlike prior works reliant on multiple off-the-shelf models, ours is a relatively unified approach and so it is resilient to errors in other off-the-shelf models. Extensive experiments on FFHQ and CelebA datasets validate the efficacy and robustness of our approach, showcasing high-fidelity, realistic face-swapping with minimal inference time. Our code is available at https://github.com/Sanoojan/REFace.