# Face Adapter for Pre-Trained Diffusion Models with Fine-Grained ID and Attribute Control

arXiv: https://arxiv.org/abs/2405.12970 · HTML: https://mlsanity.com/paper/2405.12970/

## Key facts

- Cited by 67 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 38 of its 49 references that are in the map.
- First posted on arXiv 21 May 2024.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 38)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2308.06721/) (2023)
- [FaceForensics++: Learning to Detect Manipulated Facial Images](https://mlsanity.com/paper/1901.08971/) (2019)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [First Order Motion Model for Image Animation](https://mlsanity.com/paper/2003.00196/) (2020)
- [Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation](https://mlsanity.com/paper/2311.17117/) (2023)
- [BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation](https://mlsanity.com/paper/1808.00897/) (2018)
- [Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Set](https://mlsanity.com/paper/1903.08527/) (2019)
- [One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing](https://mlsanity.com/paper/2011.15126/) (2020)
- [InstantID: Zero-shot Identity-Preserving Generation in Seconds](https://mlsanity.com/paper/2401.07519/) (2024)
- [CurricularFace: Adaptive Curriculum Learning Loss for Deep Face Recognition](https://mlsanity.com/paper/2004.00288/) (2020)
- [FSGAN: Subject Agnostic Face Swapping and Reenactment](https://mlsanity.com/paper/1908.05932/) (2019)
- [FastComposer: Tuning-Free Multi-Subject Image Generation with Localized Attention](https://mlsanity.com/paper/2305.10431/) (2023)
- [SimSwap: An Efficient Framework For High Fidelity Face Swapping](https://mlsanity.com/paper/2106.06340/) (2021)
- [FaceShifter: Towards High Fidelity And Occlusion Aware Face Swapping](https://mlsanity.com/paper/1912.13457/) (2019)
- [Thin-Plate Spline Motion Model for Image Animation](https://mlsanity.com/paper/2203.14367/) (2022)
- [Animating Arbitrary Objects via Deep Motion Transfer](https://mlsanity.com/paper/1812.08861/) (2018)
- [PIRenderer: Controllable Portrait Image Generation via Semantic Neural Rendering](https://mlsanity.com/paper/2109.08379/) (2021)
- [Depth-Aware Generative Adversarial Network for Talking Head Video Generation](https://mlsanity.com/paper/2203.06605/) (2022)
- [StyleHEAT: One-Shot High-Resolution Editable Talking Face Generation via Pre-trained StyleGAN](https://mlsanity.com/paper/2203.04036/) (2022)
- [One Shot Face Swapping on Megapixels](https://mlsanity.com/paper/2105.04932/) (2021)
- [HifiFace: 3D Shape and Semantic Prior Guided High Fidelity Face Swapping](https://mlsanity.com/paper/2106.09965/) (2021)
- [PortraitBooth: A Versatile Portrait Model for Fast Identity-preserved Personalization](https://mlsanity.com/paper/2312.06354/) (2023)
- [MetaPortrait: Identity-Preserving Talking Head Generation with Fast Personalized Adaptation](https://mlsanity.com/paper/2212.08062/) (2022)
- [Fine-Grained Face Swapping via Regional GAN Inversion](https://mlsanity.com/paper/2211.14068/) (2022)
- [BlendFace: Re-designing Identity Encoders for Face-Swapping](https://mlsanity.com/paper/2307.10854/) (2023)
- [Implicit Identity Representation Conditioned Memory Compensation Network for Talking Head video Generation](https://mlsanity.com/paper/2307.09906/) (2023)

## Built on by (most cited first, 30 of 67)

- [LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control](https://mlsanity.com/paper/2407.03168/) (2024)
- [MotionBooth: Motion-Aware Customized Text-to-Video Generation](https://mlsanity.com/paper/2406.17758/) (2024)
- [DreamVideo-2: Zero-Shot Subject-Driven Video Customization with Precise Motion Control](https://mlsanity.com/paper/2410.13830/) (2024)
- [Stand-In: A Lightweight and Plug-and-Play Identity Control for Video Generation](https://mlsanity.com/paper/2508.07901/) (2025)
- [DiffSensei: Bridging Multi-Modal LLMs and Diffusion Models for Customized Manga Generation](https://mlsanity.com/paper/2412.07589/) (2024)
- [VideoMaker: Zero-shot Customized Video Generation with the Inherent Force of Video Diffusion Models](https://mlsanity.com/paper/2412.19645/) (2024)
- [CanonSwap: High-Fidelity and Consistent Video Face Swapping via Canonical Space Modulation](https://mlsanity.com/paper/2507.02691/) (2025)
- [AdvI2I: Adversarial Image Attack on Image-to-Image Diffusion models](https://mlsanity.com/paper/2410.21471/) (2024)
- [DynamicFace: High-Quality and Consistent Face Swapping for Image and Video using Composable 3D Facial Priors](https://mlsanity.com/paper/2501.08553/) (2025)
- [DynamicID: Zero-Shot Multi-ID Image Personalization with Flexible Facial Editability](https://mlsanity.com/paper/2503.06505/) (2025)
- [DreamID: High-Fidelity and Fast diffusion-based Face Swapping via Triplet ID Group Learning](https://mlsanity.com/paper/2504.14509/) (2025)
- [Text-to-Image Synthesis: A Decade Survey](https://mlsanity.com/paper/2411.16164/) (2024)
- [HiFiVFS: High Fidelity Video Face Swapping](https://mlsanity.com/paper/2411.18293/) (2024)
- [PersonaLive! Expressive Portrait Image Animation for Live Streaming](https://mlsanity.com/paper/2512.11253/) (2025)
- [DreamID-V:Bridging the Image-to-Video Gap for High-Fidelity Face Swapping via Diffusion Transformer](https://mlsanity.com/paper/2601.01425/) (2026)
- [MagicFace: Training-free Universal-Style Human Image Customized Synthesis](https://mlsanity.com/paper/2408.07433/) (2024)
- [MagicFace: High-Fidelity Facial Expression Editing with Action-Unit Control](https://mlsanity.com/paper/2501.02260/) (2025)
- [High-Fidelity Diffusion Face Swapping with ID-Constrained Facial Conditioning](https://mlsanity.com/paper/2503.22179/) (2025)
- [My Face Is Mine, Not Yours: Facial Protection Against Diffusion Model Face Swapping](https://mlsanity.com/paper/2505.15336/) (2025)
- [FlashPortrait: 6x Faster Infinite Portrait Animation with Adaptive Latent Prediction](https://mlsanity.com/paper/2512.16900/) (2025)
- [Auto Cherry-Picker: Learning from High-quality Generative Data Driven by Language](https://mlsanity.com/paper/2406.20085/) (2024)
- [EmojiDiff: Advanced Facial Expression Control with High Identity Preservation in Portrait Generation](https://mlsanity.com/paper/2412.01254/) (2024)
- [ArtCrafter: Text-Image Aligning Style Transfer via Embedding Reframing](https://mlsanity.com/paper/2501.02064/) (2025)
- [InterAnimate: Taming Region-aware Diffusion Model for Realistic Human Interaction Animation](https://mlsanity.com/paper/2504.10905/) (2025)
- [MagicPortrait: Temporally Consistent Face Reenactment with 3D Geometric Guidance](https://mlsanity.com/paper/2504.21497/) (2025)
- [MultiCrafter: High-Fidelity Multi-Subject Generation via Disentangled Attention and Identity-Aware Preference Alignment](https://mlsanity.com/paper/2509.21953/) (2025)
- [TriDF: Evaluating Perception, Detection, and Hallucination for Interpretable DeepFake Detection](https://mlsanity.com/paper/2512.10652/) (2025)
- [DreamVideo-Omni: Omni-Motion Controlled Multi-Subject Video Customization with Latent Identity Reinforcement Learning](https://mlsanity.com/paper/2603.12257/) (2026)
- [SerialGen: Personalized Image Generation by First Standardization Then Personalization](https://mlsanity.com/paper/2412.01485/) (2024)
- [Towards Consistent and Controllable Image Synthesis for Face Editing](https://mlsanity.com/paper/2502.02465/) (2025)

## Abstract (arXiv)

Current face reenactment and swapping methods mainly rely on GAN frameworks, but recent focus has shifted to pre-trained diffusion models for their superior generation capabilities. However, training these models is resource-intensive, and the results have not yet achieved satisfactory performance levels. To address this issue, we introduce Face-Adapter, an efficient and effective adapter designed for high-precision and high-fidelity face editing for pre-trained diffusion models. We observe that both face reenactment/swapping tasks essentially involve combinations of target structure, ID and attribute. We aim to sufficiently decouple the control of these factors to achieve both tasks in one model. Specifically, our method contains: 1) A Spatial Condition Generator that provides precise landmarks and background; 2) A Plug-and-play Identity Encoder that transfers face embeddings to the text space by a transformer decoder. 3) An Attribute Controller that integrates spatial conditions and detailed attributes. Face-Adapter achieves comparable or even superior performance in terms of motion control precision, ID retention capability, and generation quality compared to fully fine-tuned face reenactment/swapping models. Additionally, Face-Adapter seamlessly integrates with various StableDiffusion models.