# AnyFace: Free-style Text-to-Face Synthesis and Manipulation

arXiv: https://arxiv.org/abs/2203.15334 · HTML: https://mlsanity.com/paper/2203.15334/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 37 references that are in the map.
- First posted on arXiv 29 Mar 2022.

## Builds on (most cited first, 27 of 27)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Training Generative Adversarial Networks with Limited Data](https://mlsanity.com/paper/2006.06676/) (2020)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation](https://mlsanity.com/paper/2008.00951/) (2020)
- [AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks](https://mlsanity.com/paper/1711.10485/) (2017)
- [NVAE: A Deep Hierarchical Variational Autoencoder](https://mlsanity.com/paper/2007.03898/) (2020)
- [Deep Mutual Learning](https://mlsanity.com/paper/1706.00384/) (2017)
- [MaskGAN: Towards Diverse and Interactive Facial Image Manipulation](https://mlsanity.com/paper/1907.11922/) (2019)
- [Designing an Encoder for StyleGAN Image Manipulation](https://mlsanity.com/paper/2102.02766/) (2021)
- [DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesis](https://mlsanity.com/paper/1904.01310/) (2019)
- [Controllable Text-to-Image Generation](https://mlsanity.com/paper/1909.07083/) (2019)
- [ManiGAN: Text-Guided Image Manipulation](https://mlsanity.com/paper/1912.06203/) (2019)
- [One Shot Face Swapping on Megapixels](https://mlsanity.com/paper/2105.04932/) (2021)
- [Text-Adaptive Generative Adversarial Networks: Manipulating Images with Natural Language](https://mlsanity.com/paper/1810.11919/) (2018)
- [Semantic Image Synthesis via Adversarial Learning](https://mlsanity.com/paper/1707.06873/) (2017)
- [Cycle-Consistent Inverse GAN for Text-to-Image Synthesis](https://mlsanity.com/paper/2108.01361/) (2021)
- [Towards Open-World Text-Guided Face Image Generation and Manipulation](https://mlsanity.com/paper/2104.08910/) (2021)
- [Conditional Image Generation and Manipulation for User-Specified Content](https://mlsanity.com/paper/2005.04909/) (2020)
- [CookGAN: Meal Image Synthesis from Ingredients](https://mlsanity.com/paper/2002.11493/) (2020)
- [Text2FaceGAN: Face Generation from Fine Grained Textual Descriptions](https://mlsanity.com/paper/1911.11378/) (2019)
- [Reference-guided Face Component Editing](https://mlsanity.com/paper/2006.02051/) (2020)

## Built on by (most cited first, 30 of 33)

- [Collaborative Diffusion for Multi-Modal Face Generation and Editing](https://mlsanity.com/paper/2304.10530/) (2023)
- [TALL: Thumbnail Layout for Deepfake Video Detection](https://mlsanity.com/paper/2307.07494/) (2023)
- [Deepfake Generation and Detection: A Benchmark and Survey](https://mlsanity.com/paper/2403.17881/) (2024)
- [Text2Performer: Text-Driven Human Video Generation](https://mlsanity.com/paper/2304.08483/) (2023)
- [CLIPVG: Text-Guided Image Manipulation Using Differentiable Vector Graphics](https://mlsanity.com/paper/2212.02122/) (2022)
- [High-Fidelity 3D Face Generation from Natural Language Descriptions](https://mlsanity.com/paper/2305.03302/) (2023)
- [TalkCLIP: Talking Head Generation with Text-Guided Expressive Speaking Styles](https://mlsanity.com/paper/2304.00334/) (2023)
- [DeltaEdit: Exploring Text-free Training for Text-Driven Image Manipulation](https://mlsanity.com/paper/2303.06285/) (2023)
- [Catch Missing Details: Image Reconstruction with Frequency Augmented Variational Autoencoder](https://mlsanity.com/paper/2305.02541/) (2023)
- [Pluralistic Aging Diffusion Autoencoder](https://mlsanity.com/paper/2303.11086/) (2023)
- [GAN-based Facial Attribute Manipulation](https://mlsanity.com/paper/2210.12683/) (2022)
- [Bridging CLIP and StyleGAN through Latent Alignment for Image Editing](https://mlsanity.com/paper/2210.04506/) (2022)
- [Towards High-Fidelity Text-Guided 3D Face Generation and Manipulation Using only Images](https://mlsanity.com/paper/2308.16758/) (2023)
- [Multimodal Dialog Systems with Dual Knowledge-enhanced Generative Pretrained Language Model](https://mlsanity.com/paper/2207.07934/) (2022)
- [HumanDiffusion: a Coarse-to-Fine Alignment Diffusion Framework for Controllable Text-Driven Person Image Generation](https://mlsanity.com/paper/2211.06235/) (2022)
- [Text-guided Eyeglasses Manipulation with Spatial Constraints](https://mlsanity.com/paper/2304.12539/) (2023)
- [InstructPix2NeRF: Instructed 3D Portrait Editing from a Single Image](https://mlsanity.com/paper/2311.02826/) (2023)
- [Vision + Language Applications: A Survey](https://mlsanity.com/paper/2305.14598/) (2023)
- [Fast Text-to-3D-Aware Face Generation and Manipulation via Direct Cross-modal Mapping and Geometric Regularization](https://mlsanity.com/paper/2403.06702/) (2024)
- [Efficient Text-Guided 3D-Aware Portrait Generation with Score Distillation Sampling on Distribution](https://mlsanity.com/paper/2306.02083/) (2023)
- [DeltaSpace: A Semantic-aligned Feature Space for Flexible Text-guided Image Editing](https://mlsanity.com/paper/2310.08785/) (2023)
- [TextGaze: Gaze-Controllable Face Generation with Natural Language](https://mlsanity.com/paper/2404.17486/) (2024)
- [Improving Face Recognition from Caption Supervision with Multi-Granular Contextual Feature Aggregation](https://mlsanity.com/paper/2308.06866/) (2023)
- [Text-Free Learning of a Natural Language Interface for Pretrained Face Generators](https://mlsanity.com/paper/2209.03953/) (2022)
- [Robust Text-driven Image Editing Method that Adaptively Explores Directions in Latent Spaces of StyleGAN and CLIP](https://mlsanity.com/paper/2304.00964/) (2023)
- [Text-Guided Face Recognition using Multi-Granularity Cross-Modal Contrastive Learning](https://mlsanity.com/paper/2312.09367/) (2023)
- [SemFaceEdit: Semantic Face Editing on Generative Radiance Manifolds](https://mlsanity.com/paper/2506.22833/) (2025)
- [GPTFace: Generative Pre-training of Facial-Linguistic Transformer by Span Masking and Weakly Correlated Text-image Data](https://mlsanity.com/paper/2510.18345/) (2025)
- [Towards Arbitrary Text-driven Image Manipulation via Space Alignment](https://mlsanity.com/paper/2301.10670/) (2023)
- [UP-FacE: User-predictable Fine-grained Face Shape Editing](https://mlsanity.com/paper/2403.13972/) (2024)

## Abstract (arXiv)

Existing text-to-image synthesis methods generally are only applicable to words in the training dataset. However, human faces are so variable to be described with limited words. So this paper proposes the first free-style text-to-face method namely AnyFace enabling much wider open world applications such as metaverse, social media, cosmetics, forensics, etc. AnyFace has a novel two-stream framework for face image synthesis and manipulation given arbitrary descriptions of the human face. Specifically, one stream performs text-to-face generation and the other conducts face image reconstruction. Facial text and image features are extracted using the CLIP (Contrastive Language-Image Pre-training) encoders. And a collaborative Cross Modal Distillation (CMD) module is designed to align the linguistic and visual features across these two streams. Furthermore, a Diverse Triplet Loss (DT loss) is developed to model fine-grained features and improve facial diversity. Extensive experiments on Multi-modal CelebA-HQ and CelebAText-HQ demonstrate significant advantages of AnyFace over state-of-the-art methods. AnyFace can achieve high-quality, high-resolution, and high-diversity face synthesis and manipulation results without any constraints on the number and content of input captions.