# Face0: Instantaneously Conditioning a Text-to-Image Model on a Face

arXiv: https://arxiv.org/abs/2306.06638 · HTML: https://mlsanity.com/paper/2306.06638/

## Key facts

- Cited by 78 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 34 references that are in the map.
- First posted on arXiv 11 Jun 2023.

## Builds on (most cited first, 29 of 29)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://mlsanity.com/paper/2206.10789/) (2022)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [VGGFace2: A dataset for recognising faces across pose and age](https://mlsanity.com/paper/1710.08092/) (2017)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [Muse: Text-To-Image Generation via Masked Generative Transformers](https://mlsanity.com/paper/2301.00704/) (2023)
- [Multimodal datasets: misogyny, pornography, and malignant stereotypes](https://mlsanity.com/paper/2110.01963/) (2021)
- [Encoder-based Domain Tuning for Fast Personalization of Text-to-Image Models](https://mlsanity.com/paper/2302.12228/) (2023)
- [Dreamix: Video Diffusion Models are General Video Editors](https://mlsanity.com/paper/2302.01329/) (2023)
- [One Shot Face Swapping on Megapixels](https://mlsanity.com/paper/2105.04932/) (2021)
- [CLIP2StyleGAN: Unsupervised Extraction of StyleGAN Edit Directions](https://mlsanity.com/paper/2112.05219/) (2021)
- [MyStyle: A Personalized Generative Prior](https://mlsanity.com/paper/2203.17272/) (2022)

## Built on by (most cited first, 30 of 78)

- [InstantID: Zero-shot Identity-Preserving Generation in Seconds](https://mlsanity.com/paper/2401.07519/) (2024)
- [HyperDreamBooth: HyperNetworks for Fast Personalization of Text-to-Image Models](https://mlsanity.com/paper/2307.06949/) (2023)
- [PuLID: Pure and Lightning ID Customization via Contrastive Alignment](https://mlsanity.com/paper/2404.16022/) (2024)
- [SSR-Encoder: Encoding Selective Subject Representation for Subject-Driven Generation](https://mlsanity.com/paper/2312.16272/) (2023)
- [DreamBench++: A Human-Aligned Benchmark for Personalized Image Generation](https://mlsanity.com/paper/2406.16855/) (2024)
- [VideoBooth: Diffusion-based Video Generation with Image Prompts](https://mlsanity.com/paper/2312.00777/) (2023)
- [ID-Animator: Zero-Shot Identity-Preserving Human Video Generation](https://mlsanity.com/paper/2404.15275/) (2024)
- [Arc2Face: A Foundation Model for ID-Consistent Human Faces](https://mlsanity.com/paper/2403.11641/) (2024)
- [Controllable Generation with Text-to-Image Diffusion Models: A Survey](https://mlsanity.com/paper/2403.04279/) (2024)
- [PortraitBooth: A Versatile Portrait Model for Fast Identity-preserved Personalization](https://mlsanity.com/paper/2312.06354/) (2023)
- [Multi-subject Open-set Personalization in Video Generation](https://mlsanity.com/paper/2501.06187/) (2025)
- [The Chosen One: Consistent Characters in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2311.10093/) (2023)
- [FaceStudio: Put Your Face Everywhere in Seconds](https://mlsanity.com/paper/2312.02663/) (2023)
- [ConsistentID: Portrait Generation with Multimodal Fine-Grained Identity Preserving](https://mlsanity.com/paper/2404.16771/) (2024)
- [Customizing Text-to-Image Models with a Single Image Pair](https://mlsanity.com/paper/2405.01536/) (2024)
- [PALP: Prompt Aligned Personalization of Text-to-Image Models](https://mlsanity.com/paper/2401.06105/) (2024)
- [IDAdapter: Learning Mixed Features for Tuning-Free Personalization of Text-to-Image Models](https://mlsanity.com/paper/2403.13535/) (2024)
- [InstantFamily: Masked Attention for Zero-shot Multi-ID Image Generation](https://mlsanity.com/paper/2404.19427/) (2024)
- [A Survey on Personalized Content Synthesis with Diffusion Models](https://mlsanity.com/paper/2405.05538/) (2024)
- [PersonalVideo: High ID-Fidelity Video Customization without Dynamic and Semantic Degradation](https://mlsanity.com/paper/2411.17048/) (2024)
- [Edify Image: High-Quality Image Generation with Pixel Space Laplacian Diffusion Models](https://mlsanity.com/paper/2411.07126/) (2024)
- [Group Diffusion Transformers are Unsupervised Multitask Learners](https://mlsanity.com/paper/2410.15027/) (2024)
- [IDProtector: An Adversarial Noise Encoder to Protect Against ID-Preserving Image Generation](https://mlsanity.com/paper/2412.11638/) (2024)
- [Nested Attention: Semantic-aware Attention Values for Concept Personalization](https://mlsanity.com/paper/2501.01407/) (2025)
- [PreciseControl: Enhancing Text-To-Image Diffusion Models with Fine-Grained Attribute Control](https://mlsanity.com/paper/2408.05083/) (2024)
- [Personalized Image Generation with Deep Generative Models: A Decade Survey](https://mlsanity.com/paper/2502.13081/) (2025)
- [ID3: Identity-Preserving-yet-Diversified Diffusion Models for Synthetic Face Recognition](https://mlsanity.com/paper/2409.17576/) (2024)
- [MagicID: Hybrid Preference Optimization for ID-Consistent and Dynamic-Preserved Video Customization](https://mlsanity.com/paper/2503.12689/) (2025)
- [WithAnyone: Towards Controllable and ID Consistent Image Generation](https://mlsanity.com/paper/2510.14975/) (2025)
- [Arc2Avatar: Generating Expressive 3D Avatars from a Single Image via ID Guidance](https://mlsanity.com/paper/2501.05379/) (2025)

## Abstract (arXiv)

We present Face0, a novel way to instantaneously condition a text-to-image generation model on a face, in sample time, without any optimization procedures such as fine-tuning or inversions. We augment a dataset of annotated images with embeddings of the included faces and train an image generation model, on the augmented dataset. Once trained, our system is practically identical at inference time to the underlying base model, and is therefore able to generate images, given a user-supplied face image and a prompt, in just a couple of seconds. Our method achieves pleasing results, is remarkably simple, extremely fast, and equips the underlying model with new capabilities, like controlling the generated images both via text or via direct manipulation of the input face embeddings. In addition, when using a fixed random vector instead of a face embedding from a user supplied image, our method essentially solves the problem of consistent character generation across images. Finally, while requiring further research, we hope that our method, which decouples the model's textual biases from its biases on faces, might be a step towards some mitigation of biases in future text-to-image models.