# MagicFace: High-Fidelity Facial Expression Editing with Action-Unit Control

arXiv: https://arxiv.org/abs/2501.02260 · HTML: https://mlsanity.com/paper/2501.02260/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 38 of its 62 references that are in the map.
- First posted on arXiv 4 Jan 2025.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 38)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation](https://mlsanity.com/paper/1711.09020/) (2017)
- [MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing](https://mlsanity.com/paper/2304.08465/) (2023)
- [Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation](https://mlsanity.com/paper/2311.17117/) (2023)
- [How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)](https://mlsanity.com/paper/1703.07332/) (2017)
- [Diffusion Autoencoders: Toward a Meaningful and Decodable Representation](https://mlsanity.com/paper/2111.15640/) (2021)
- [BiSeNet V2: Bilateral Network with Guided Aggregation for Real-time Semantic Segmentation](https://mlsanity.com/paper/2004.02147/) (2020)
- [DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editing](https://mlsanity.com/paper/2306.14435/) (2023)
- [AttGAN: Facial Attribute Editing by Only Changing What You Want](https://mlsanity.com/paper/1711.10678/) (2017)
- [Deep Affect Prediction in-the-wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyond](https://mlsanity.com/paper/1804.10938/) (2018)
- [AniPortrait: Audio-Driven Synthesis of Photorealistic Portrait Animation](https://mlsanity.com/paper/2403.17694/) (2024)
- [GANimation: Anatomically-aware Facial Animation from a Single Image](https://mlsanity.com/paper/1807.09251/) (2018)
- [OOTDiffusion: Outfitting Fusion based Latent Diffusion for Controllable Virtual Try-on](https://mlsanity.com/paper/2403.01779/) (2024)
- [X-Portrait: Expressive Portrait Animation with Hierarchical Motion Attention](https://mlsanity.com/paper/2403.15931/) (2024)
- [DiffusionRig: Learning Personalized Priors for Facial Appearance Editing](https://mlsanity.com/paper/2304.06711/) (2023)
- [Face Adapter for Pre-Trained Diffusion Models with Fine-Grained ID and Attribute Control](https://mlsanity.com/paper/2405.12970/) (2024)
- [The Blessing of Randomness: SDE Beats ODE in General Diffusion-based Image Editing](https://mlsanity.com/paper/2311.01410/) (2023)
- [Dif-Fusion: Towards High Color Fidelity in Infrared and Visible Image Fusion with Diffusion Models](https://mlsanity.com/paper/2301.08072/) (2023)
- [ICface: Interpretable and Controllable Face Reenactment Using GANs](https://mlsanity.com/paper/1904.01909/) (2019)
- [ExprGAN: Facial Expression Editing with Controllable Expression Intensity](https://mlsanity.com/paper/1709.03842/) (2017)
- [InstantDrag: Improving Interactivity in Drag-based Image Editing](https://mlsanity.com/paper/2409.08857/) (2024)
- [DeltaEdit: Exploring Text-free Training for Text-Driven Image Manipulation](https://mlsanity.com/paper/2303.06285/) (2023)
- [High Resolution Face Editing with Masked GAN Latent Code Optimization](https://mlsanity.com/paper/2103.11135/) (2021)

## Built on by (most cited first, 7 of 7)

- [DeX-Portrait: Disentangled and Expressive Portrait Animation via Explicit and Latent Motion Representations](https://mlsanity.com/paper/2512.15524/) (2025)
- [ID-Consistent, Precise Expression Generation with Blendshape-Guided Diffusion](https://mlsanity.com/paper/2510.04706/) (2025)
- [Controlled Face Manipulation and Synthesis for Data Augmentation](https://mlsanity.com/paper/2602.19219/) (2026)
- [PixelSmile: Toward Fine-Grained Facial Expression Editing](https://mlsanity.com/paper/2603.25728/) (2026)
- [Pain in 3D: Generating Controllable Synthetic Faces for Automated Pain Assessment](https://mlsanity.com/paper/2509.16727/) (2025)
- [U-Face: An Efficient and Generalizable Framework for Unsupervised Facial Attribute Editing via Subspace Learning](https://mlsanity.com/paper/2603.14004/) (2026)
- [Beyond End-to-End Black Box Mapping: An Intentional Agent Framework for Cognitive-driven Facial Reaction Generation](https://mlsanity.com/paper/2609.34419/) (2026)

## Abstract (arXiv)

We address the problem of facial expression editing by controling the relative variation of facial action-unit (AU) from the same person. This enables us to edit this specific person's expression in a fine-grained, continuous and interpretable manner, while preserving their identity, pose, background and detailed facial attributes. Key to our model, which we dub MagicFace, is a diffusion model conditioned on AU variations and an ID encoder to preserve facial details of high consistency. Specifically, to preserve the facial details with the input identity, we leverage the power of pretrained Stable-Diffusion models and design an ID encoder to merge appearance features through self-attention. To keep background and pose consistency, we introduce an efficient Attribute Controller by explicitly informing the model of current background and pose of the target. By injecting AU variations into a denoising UNet, our model can animate arbitrary identities with various AU combinations, yielding superior results in high-fidelity expression editing compared to other facial expression editing works. Code is publicly available at https://github.com/weimengting/MagicFace.