# Towards a Simultaneous and Granular Identity-Expression Control in Personalized Face Generation

arXiv: https://arxiv.org/abs/2401.01207 · HTML: https://mlsanity.com/paper/2401.01207/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 12 of its 18 references that are in the map.
- First posted on arXiv 2 Jan 2024.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 12 of 12)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Alias-Free Generative Adversarial Networks](https://mlsanity.com/paper/2106.12423/) (2021)
- [FaceForensics++: Learning to Detect Manipulated Facial Images](https://mlsanity.com/paper/1901.08971/) (2019)
- [Pseudo Numerical Methods for Diffusion Models on Manifolds](https://mlsanity.com/paper/2202.09778/) (2022)
- [A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild](https://mlsanity.com/paper/2008.10010/) (2020)
- [MaskGAN: Towards Diverse and Interactive Facial Image Manipulation](https://mlsanity.com/paper/1907.11922/) (2019)
- [Deep Video Portraits](https://mlsanity.com/paper/1805.11714/) (2018)
- [SimSwap: An Efficient Framework For High Fidelity Face Swapping](https://mlsanity.com/paper/2106.06340/) (2021)
- [FaceShifter: Towards High Fidelity And Occlusion Aware Face Swapping](https://mlsanity.com/paper/1912.13457/) (2019)
- [HifiFace: 3D Shape and Semantic Prior Guided High Fidelity Face Swapping](https://mlsanity.com/paper/2106.09965/) (2021)

## Built on by (most cited first, 26 of 26)

- [Controllable Generation with Text-to-Image Diffusion Models: A Survey](https://mlsanity.com/paper/2403.04279/) (2024)
- [Deepfake Generation and Detection: A Benchmark and Survey](https://mlsanity.com/paper/2403.17881/) (2024)
- [ConsistentID: Portrait Generation with Multimodal Fine-Grained Identity Preserving](https://mlsanity.com/paper/2404.16771/) (2024)
- [MegActor: Harness the Power of Raw Video for Vivid Portrait Animation](https://mlsanity.com/paper/2405.20851/) (2024)
- [X-NeMo: Expressive Neural Motion Reenactment via Disentangled Latent Attention](https://mlsanity.com/paper/2507.23143/) (2025)
- [Human Motion Video Generation: A Survey](https://mlsanity.com/paper/2509.03883/) (2025)
- [Personalized Image Generation with Deep Generative Models: A Decade Survey](https://mlsanity.com/paper/2502.13081/) (2025)
- [Learn2Talk: 3D Talking Face Learns from 2D Talking Face](https://mlsanity.com/paper/2404.12888/) (2024)
- [InstructAvatar: Text-Guided Emotion and Motion Control for Avatar Generation](https://mlsanity.com/paper/2405.15758/) (2024)
- [Towards Localized Fine-Grained Control for Facial Expression Generation](https://mlsanity.com/paper/2407.20175/) (2024)
- [Deepfake Media Generation and Detection in the Generative AI Era: A Survey and Outlook](https://mlsanity.com/paper/2411.19537/) (2024)
- [DynamicID: Zero-Shot Multi-ID Image Personalization with Flexible Facial Editability](https://mlsanity.com/paper/2503.06505/) (2025)
- [HiFiVFS: High Fidelity Video Face Swapping](https://mlsanity.com/paper/2411.18293/) (2024)
- [GAITGen: Disentangled Motion-Pathology Impaired Gait Generative Model -- Bringing Motion Generation to the Clinical Domain](https://mlsanity.com/paper/2503.22397/) (2025)
- [DNPM: A Neural Parametric Model for the Synthesis of Facial Geometric Details](https://mlsanity.com/paper/2405.19688/) (2024)
- [EmojiDiff: Advanced Facial Expression Control with High Identity Preservation in Portrait Generation](https://mlsanity.com/paper/2412.01254/) (2024)
- [Dense-Face: Personalized Face Generation Model via Dense Annotation Prediction](https://mlsanity.com/paper/2412.18149/) (2024)
- [Z-Magic: Zero-shot Multiple Attributes Guided Image Creator](https://mlsanity.com/paper/2503.12124/) (2025)
- [MasterWeaver: Taming Editability and Face Identity for Personalized Text-to-Image Generation](https://mlsanity.com/paper/2405.05806/) (2024)
- [ID-Consistent, Precise Expression Generation with Blendshape-Guided Diffusion](https://mlsanity.com/paper/2510.04706/) (2025)
- [Reverse Personalization](https://mlsanity.com/paper/2512.22984/) (2025)
- [CustomTex: High-fidelity Indoor Scene Texturing via Multi-Reference Customization](https://mlsanity.com/paper/2603.19121/) (2026)
- [GenEAva: Generating Cartoon Avatars with Fine-Grained Facial Expressions from Realistic Diffusion-based Faces](https://mlsanity.com/paper/2504.07945/) (2025)
- [Gen-AFFECT: Generation of Avatar Fine-grained Facial Expressions with Consistent identiTy](https://mlsanity.com/paper/2508.09461/) (2025)
- [Instant Expressive Gaussian Head Avatars at Over 100 FPS](https://mlsanity.com/paper/2512.16893/) (2025)
- [GaussianSwap: Animatable Video Face Swapping with 3D Gaussian Splatting](https://mlsanity.com/paper/2601.05511/) (2026)

## Abstract (arXiv)

In human-centric content generation, the pre-trained text-to-image models struggle to produce user-wanted portrait images, which retain the identity of individuals while exhibiting diverse expressions. This paper introduces our efforts towards personalized face generation. To this end, we propose a novel multi-modal face generation framework, capable of simultaneous identity-expression control and more fine-grained expression synthesis. Our expression control is so sophisticated that it can be specialized by the fine-grained emotional vocabulary. We devise a novel diffusion model that can undertake the task of simultaneously face swapping and reenactment. Due to the entanglement of identity and expression, it's nontrivial to separately and precisely control them in one framework, thus has not been explored yet. To overcome this, we propose several innovative designs in the conditional diffusion model, including balancing identity and expression encoder, improved midpoint sampling, and explicitly background conditioning. Extensive experiments have demonstrated the controllability and scalability of the proposed framework, in comparison with state-of-the-art text-to-image, face swapping, and face reenactment methods.