# Towards Open-World Text-Guided Face Image Generation and Manipulation

arXiv: https://arxiv.org/abs/2104.08910 · HTML: https://mlsanity.com/paper/2104.08910/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 54 of its 73 references that are in the map.
- First posted on arXiv 18 Apr 2021.

## Builds on (most cited first, 30 of 54)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop](https://mlsanity.com/paper/1506.03365/) (2015)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [Generative Adversarial Text to Image Synthesis](https://mlsanity.com/paper/1605.05396/) (2016)
- [Interpreting the Latent Space of GANs for Semantic Face Editing](https://mlsanity.com/paper/1907.10786/) (2019)
- [AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks](https://mlsanity.com/paper/1711.10485/) (2017)
- [MaskGAN: Towards Diverse and Interactive Facial Image Manipulation](https://mlsanity.com/paper/1907.11922/) (2019)
- [StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks](https://mlsanity.com/paper/1612.03242/) (2016)
- [In-Domain GAN Inversion for Real Image Editing](https://mlsanity.com/paper/2004.00049/) (2020)
- [Stacked Cross Attention for Image-Text Matching](https://mlsanity.com/paper/1803.08024/) (2018)
- [GAN Inversion: A Survey](https://mlsanity.com/paper/2101.05278/) (2021)
- [Image2StyleGAN++: How to Edit the Embedded Images?](https://mlsanity.com/paper/1911.11544/) (2019)
- [Towards Real-World Blind Face Restoration with Generative Facial Prior](https://mlsanity.com/paper/2101.04061/) (2021)
- [DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesis](https://mlsanity.com/paper/1904.01310/) (2019)
- [SEAN: Image Synthesis with Semantic Region-Adaptive Normalization](https://mlsanity.com/paper/1911.12861/) (2019)
- [StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networks](https://mlsanity.com/paper/1710.10916/) (2017)
- [Exploiting Deep Generative Prior for Versatile Image Restoration and Manipulation](https://mlsanity.com/paper/2003.13659/) (2020)

## Built on by (most cited first, 25 of 25)

- [Blended Diffusion for Text-driven Editing of Natural Images](https://mlsanity.com/paper/2111.14818/) (2021)
- [Collaborative Diffusion for Multi-Modal Face Generation and Editing](https://mlsanity.com/paper/2304.10530/) (2023)
- [Unified Multi-Modal Latent Diffusion for Joint Subject and Text Conditional Image Generation](https://mlsanity.com/paper/2303.09319/) (2023)
- [Predict, Prevent, and Evaluate: Disentangled Text-Driven Image Manipulation Empowered by Pre-Trained Vision-Language Model](https://mlsanity.com/paper/2111.13333/) (2021)
- [AnyFace: Free-style Text-to-Face Synthesis and Manipulation](https://mlsanity.com/paper/2203.15334/) (2022)
- [StyleT2I: Toward Compositional and High-Fidelity Text-to-Image Synthesis](https://mlsanity.com/paper/2203.15799/) (2022)
- [TalkCLIP: Talking Head Generation with Text-Guided Expressive Speaking Styles](https://mlsanity.com/paper/2304.00334/) (2023)
- [An Image is Worth More Than a Thousand Words: Towards Disentanglement in the Wild](https://mlsanity.com/paper/2106.15610/) (2021)
- [Local 3D Editing via 3D Distillation of CLIP Knowledge](https://mlsanity.com/paper/2306.12570/) (2023)
- [Attribute-Centric Compositional Text-to-Image Generation](https://mlsanity.com/paper/2301.01413/) (2023)
- [Deep Face Restoration: A Survey](https://mlsanity.com/paper/2211.02831/) (2022)
- [A Survey on Deep Generative 3D-aware Image Synthesis](https://mlsanity.com/paper/2210.14267/) (2022)
- [DE-Net: Dynamic Text-guided Image Editing Adversarial Networks](https://mlsanity.com/paper/2206.01160/) (2022)
- [Towards High-Fidelity Text-Guided 3D Face Generation and Manipulation Using only Images](https://mlsanity.com/paper/2308.16758/) (2023)
- [Synthesize Privacy-Preserving High-Resolution Images via Private Textual Intermediaries](https://mlsanity.com/paper/2506.07555/) (2025)
- [AI Illustrator: Translating Raw Descriptions into Images by Prompt-based Cross-Modal Generation](https://mlsanity.com/paper/2209.03160/) (2022)
- [Multimodality-guided Image Style Transfer using Cross-modal GAN Inversion](https://mlsanity.com/paper/2312.01671/) (2023)
- [Diffusion-based Facial Aesthetics Enhancement with 3D Structure Guidance](https://mlsanity.com/paper/2503.14402/) (2025)
- [LLV-FSR: Exploiting Large Language-Vision Prior for Face Super-resolution](https://mlsanity.com/paper/2411.09293/) (2024)
- [Target-Free Text-guided Image Manipulation](https://mlsanity.com/paper/2211.14544/) (2022)
- [GPTFace: Generative Pre-training of Facial-Linguistic Transformer by Span Masking and Weakly Correlated Text-image Data](https://mlsanity.com/paper/2510.18345/) (2025)
- [ProxT2I: Efficient Reward-Guided Text-to-Image Generation via Proximal Diffusion](https://mlsanity.com/paper/2511.18742/) (2025)
- [Modelling Latent Dynamics of StyleGAN using Neural ODEs](https://mlsanity.com/paper/2208.11197/) (2022)
- [TextCLIP: Text-Guided Face Image Generation And Manipulation Without Adversarial Training](https://mlsanity.com/paper/2309.11923/) (2023)
- [MMFace-DiT: A Dual-Stream Diffusion Transformer for High-Fidelity Multimodal Face Generation](https://mlsanity.com/paper/2603.29029/) (2026)

## Abstract (arXiv)

The existing text-guided image synthesis methods can only produce limited quality results with at most mbox2562 resolution and the textual instructions are constrained in a small Corpus. In this work, we propose a unified framework for both face image generation and manipulation that produces diverse and high-quality images with an unprecedented resolution at 1024 from multimodal inputs. More importantly, our method supports open-world scenarios, including both image and text, without any re-training, fine-tuning, or post-processing. To be specific, we propose a brand new paradigm of text-guided image generation and manipulation based on the superior characteristics of a pretrained GAN model. Our proposed paradigm includes two novel strategies. The first strategy is to train a text encoder to obtain latent codes that align with the hierarchically semantic of the aforementioned pretrained GAN model. The second strategy is to directly optimize the latent codes in the latent space of the pretrained GAN model with guidance from a pretrained language model. The latent codes can be randomly sampled from a prior distribution or inverted from a given image, which provides inherent supports for both image generation and manipulation from multi-modal inputs, such as sketches or semantic labels, with textual guidance. To facilitate text-guided multi-modal synthesis, we propose the Multi-Modal CelebA-HQ, a large-scale dataset consisting of real face images and corresponding semantic segmentation map, sketch, and textual descriptions. Extensive experiments on the introduced dataset demonstrate the superior performance of our proposed method. Code and data are available at https://github.com/weihaox/TediGAN.