# VividFace: A Diffusion-Based Hybrid Framework for High-Fidelity Video Face Swapping

arXiv: https://arxiv.org/abs/2412.11279 · HTML: https://mlsanity.com/paper/2412.11279/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 50 references that are in the map.
- First posted on arXiv 15 Dec 2024.

## Builds on (most cited first, 30 of 36)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning](https://mlsanity.com/paper/2307.04725/) (2023)
- [Imagic: Text-Based Real Image Editing with Diffusion Models](https://mlsanity.com/paper/2210.09276/) (2022)
- [Face2Face: Real-time Face Capture and Reenactment of RGB Videos](https://mlsanity.com/paper/2007.14808/) (2020)
- [MaskGAN: Towards Diverse and Interactive Facial Image Manipulation](https://mlsanity.com/paper/1907.11922/) (2019)
- [Latte: Latent Diffusion Transformer for Video Generation](https://mlsanity.com/paper/2401.03048/) (2024)
- [Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Set](https://mlsanity.com/paper/1903.08527/) (2019)
- [FSGAN: Subject Agnostic Face Swapping and Reenactment](https://mlsanity.com/paper/1908.05932/) (2019)
- [DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editing](https://mlsanity.com/paper/2306.14435/) (2023)
- [SimSwap: An Efficient Framework For High Fidelity Face Swapping](https://mlsanity.com/paper/2106.06340/) (2021)
- [SinSR: Diffusion-Based Image Super-Resolution in a Single Step](https://mlsanity.com/paper/2311.14760/) (2023)
- [EMO: Emote Portrait Alive -- Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditions](https://mlsanity.com/paper/2402.17485/) (2024)
- [FaceShifter: Towards High Fidelity And Occlusion Aware Face Swapping](https://mlsanity.com/paper/1912.13457/) (2019)
- [Implicit Diffusion Models for Continuous Super-Resolution](https://mlsanity.com/paper/2303.16491/) (2023)
- [PIRenderer: Controllable Portrait Image Generation via Semantic Neural Rendering](https://mlsanity.com/paper/2109.08379/) (2021)
- [DeepFaceLab: Integrated, flexible and extensible face-swapping framework](https://mlsanity.com/paper/2005.05535/) (2020)
- [HOPE-Net: A Graph-based Model for Hand-Object Pose Estimation](https://mlsanity.com/paper/2004.00060/) (2020)
- [One Shot Face Swapping on Megapixels](https://mlsanity.com/paper/2105.04932/) (2021)
- [HifiFace: 3D Shape and Semantic Prior Guided High Fidelity Face Swapping](https://mlsanity.com/paper/2106.09965/) (2021)
- [V-Express: Conditional Dropout for Progressive Training of Portrait Video Generation](https://mlsanity.com/paper/2406.02511/) (2024)
- [DiffFace: Diffusion-based Face Swapping with Facial Guidance](https://mlsanity.com/paper/2212.13344/) (2022)

## Built on by (most cited first, 6 of 6)

- [DrivingGen: A Comprehensive Benchmark for Generative Video World Models in Autonomous Driving](https://mlsanity.com/paper/2601.01528/) (2026)
- [CanonSwap: High-Fidelity and Consistent Video Face Swapping via Canonical Space Modulation](https://mlsanity.com/paper/2507.02691/) (2025)
- [DreamID-V:Bridging the Image-to-Video Gap for High-Fidelity Face Swapping via Diffusion Transformer](https://mlsanity.com/paper/2601.01425/) (2026)
- [Towards High Fidelity Face Swapping: A Comprehensive Survey and New Benchmark](https://mlsanity.com/paper/2605.00883/) (2026)
- [GaussianSwap: Animatable Video Face Swapping with 3D Gaussian Splatting](https://mlsanity.com/paper/2601.05511/) (2026)
- [Adaptive Identity Anchoring: Closed-Loop Keyframe Placement for Synthetic Paired Supervision in Video Face Swapping](https://mlsanity.com/paper/2607.21434/) (2026)

## Abstract (arXiv)

Video face swapping is becoming increasingly popular across various applications, yet existing methods primarily focus on static images and struggle with video face swapping because of temporal consistency and complex scenarios. In this paper, we present the first diffusion-based framework specifically designed for video face swapping. Our approach introduces a novel image-video hybrid training framework that leverages both abundant static image data and temporal video sequences, addressing the inherent limitations of video-only training. The framework incorporates a specially designed diffusion model coupled with a VidFaceVAE that effectively processes both types of data to better maintain temporal coherence of the generated videos. To further disentangle identity and pose features, we construct the Attribute-Identity Disentanglement Triplet (AIDT) Dataset, where each triplet has three face images, with two images sharing the same pose and two sharing the same identity. Enhanced with a comprehensive occlusion augmentation, this dataset also improves robustness against occlusions. Additionally, we integrate 3D reconstruction techniques as input conditioning to our network for handling large pose variations. Extensive experiments demonstrate that our framework achieves superior performance in identity preservation, temporal consistency, and visual quality compared to existing methods, while requiring fewer inference steps. Our approach effectively mitigates key challenges in video face swapping, including temporal flickering, identity preservation, and robustness to occlusions and pose variations.