# Text2Human: Text-Driven Controllable Human Image Generation

arXiv: https://arxiv.org/abs/2205.15996 · HTML: https://mlsanity.com/paper/2205.15996/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 52 of its 63 references that are in the map.
- First posted on arXiv 31 May 2022.

## Builds on (most cited first, 30 of 52)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [Alias-Free Generative Adversarial Networks](https://mlsanity.com/paper/2106.12423/) (2021)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [MaskGIT: Masked Generative Image Transformer](https://mlsanity.com/paper/2202.04200/) (2022)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [Conditional Image Generation with PixelCNN Decoders](https://mlsanity.com/paper/1606.05328/) (2016)
- [Vector Quantized Diffusion Model for Text-to-Image Synthesis](https://mlsanity.com/paper/2111.14822/) (2021)
- [AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks](https://mlsanity.com/paper/1711.10485/) (2017)
- [DensePose: Dense Human Pose Estimation In The Wild](https://mlsanity.com/paper/1802.00434/) (2018)
- [PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications](https://mlsanity.com/paper/1701.05517/) (2017)
- [Autoencoding beyond pixels using a learned similarity metric](https://mlsanity.com/paper/1512.09300/) (2015)
- [StyleFlow: Attribute-conditioned Exploration of StyleGAN-Generated Images using Conditional Continuous Normalizing Flows](https://mlsanity.com/paper/2008.02401/) (2020)
- [Everybody Dance Now](https://mlsanity.com/paper/1808.07371/) (2018)
- [Pose Guided Person Image Generation](https://mlsanity.com/paper/1705.09368/) (2017)
- [Controllable Person Image Synthesis with Attribute-Decomposed GAN](https://mlsanity.com/paper/2003.12267/) (2020)
- [StyleGAN-Human: A Data-Centric Odyssey of Human Generation](https://mlsanity.com/paper/2204.11823/) (2022)
- [PixelSNAIL: An Improved Autoregressive Generative Model](https://mlsanity.com/paper/1712.09763/) (2017)

## Built on by (most cited first, 30 of 31)

- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [Conditional Image-to-Video Generation with Latent Flow Diffusion Models](https://mlsanity.com/paper/2303.13744/) (2023)
- [LaDI-VTON: Latent Diffusion Textual-Inversion Enhanced Virtual Try-On](https://mlsanity.com/paper/2305.13501/) (2023)
- [Collaborative Diffusion for Multi-Modal Face Generation and Editing](https://mlsanity.com/paper/2304.10530/) (2023)
- [ReVersion: Diffusion-Based Relation Inversion from Images](https://mlsanity.com/paper/2303.13495/) (2023)
- [VBench++: Comprehensive and Versatile Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2411.13503/) (2024)
- [Diffusion Facial Forgery Detection](https://mlsanity.com/paper/2401.15859/) (2024)
- [HumanGen: Generating Human Radiance Fields with Explicit Priors](https://mlsanity.com/paper/2212.05321/) (2022)
- [Garment3DGen: 3D Garment Stylization and Texture Generation](https://mlsanity.com/paper/2403.18816/) (2024)
- [Text2Light: Zero-Shot Text-Driven HDR Panorama Generation](https://mlsanity.com/paper/2209.09898/) (2022)
- [FICE: Text-Conditioned Fashion Image Editing With Guided GAN Inversion](https://mlsanity.com/paper/2301.02110/) (2023)
- [Mixture of Experts (MoE): A Big Data Perspective](https://mlsanity.com/paper/2501.16352/) (2025)
- [Zero-Shot Text-to-Parameter Translation for Game Character Auto-Creation](https://mlsanity.com/paper/2303.01311/) (2023)
- [Scene Text Image Super-resolution based on Text-conditional Diffusion Models](https://mlsanity.com/paper/2311.09759/) (2023)
- [GD-VDM: Generated Depth for better Diffusion-based Video Generation](https://mlsanity.com/paper/2306.11173/) (2023)
- [StraIT: Non-autoregressive Generation with Stratified Image Transformer](https://mlsanity.com/paper/2303.00750/) (2023)
- [HandsOff: Labeled Dataset Generation With No Additional Human Annotations](https://mlsanity.com/paper/2212.12645/) (2022)
- [Vision + Language Applications: A Survey](https://mlsanity.com/paper/2305.14598/) (2023)
- [HumanVLM: Foundation for Human-Scene Vision-Language Model](https://mlsanity.com/paper/2411.03034/) (2024)
- [Multimodal-Conditioned Latent Diffusion Models for Fashion Image Editing](https://mlsanity.com/paper/2403.14828/) (2024)
- [Multimodal Benchmarking and Recommendation of Text-to-Image Generation Models](https://mlsanity.com/paper/2505.04650/) (2025)
- [Combining human parsing with analytical feature extraction and ranking schemes for high-generalization person reidentification](https://mlsanity.com/paper/2207.14243/) (2022)
- [Synthesizing Anyone, Anywhere, in Any Pose](https://mlsanity.com/paper/2304.03164/) (2023)
- [Interactive Fashion Content Generation Using LLMs and Latent Diffusion Models](https://mlsanity.com/paper/2306.05182/) (2023)
- [TexControl: Sketch-Based Two-Stage Fashion Image Generation Using Diffusion Model](https://mlsanity.com/paper/2405.04675/) (2024)
- [AnyDesign: Versatile Area Fashion Editing via Mask-Free Diffusion](https://mlsanity.com/paper/2408.11553/) (2024)
- [Dressing the Imagination: A Dataset for AI-Powered Translation of Text into Fashion Outfits and A Novel NeRA Adapter for Enhanced Feature Adaptation](https://mlsanity.com/paper/2411.13901/) (2024)
- [Zero-Shot Product Attribute Labeling with Vision-Language Models: A Three-Tier Evaluation Framework](https://mlsanity.com/paper/2601.15711/) (2026)
- [FashionSD-X: Multimodal Fashion Garment Synthesis using Latent Diffusion](https://mlsanity.com/paper/2404.18591/) (2024)
- [DPDEdit: Detail-Preserved Diffusion Models for Multimodal Fashion Image Editing](https://mlsanity.com/paper/2409.01086/) (2024)

## Abstract (arXiv)

Generating high-quality and diverse human images is an important yet challenging task in vision and graphics. However, existing generative models often fall short under the high diversity of clothing shapes and textures. Furthermore, the generation process is even desired to be intuitively controllable for layman users. In this work, we present a text-driven controllable framework, Text2Human, for a high-quality and diverse human generation. We synthesize full-body human images starting from a given human pose with two dedicated steps. 1) With some texts describing the shapes of clothes, the given human pose is first translated to a human parsing map. 2) The final human image is then generated by providing the system with more attributes about the textures of clothes. Specifically, to model the diversity of clothing textures, we build a hierarchical texture-aware codebook that stores multi-scale neural representations for each type of texture. The codebook at the coarse level includes the structural representations of textures, while the codebook at the fine level focuses on the details of textures. To make use of the learned hierarchical codebook to synthesize desired images, a diffusion-based transformer sampler with mixture of experts is firstly employed to sample indices from the coarsest level of the codebook, which then is used to predict the indices of the codebook at finer levels. The predicted indices at different levels are translated to human images by the decoder learned accompanied with hierarchical codebooks. The use of mixture-of-experts allows for the generated image conditioned on the fine-grained text input. The prediction for finer level indices refines the quality of clothing textures. Extensive quantitative and qualitative evaluations demonstrate that our proposed framework can generate more diverse and realistic human images compared to state-of-the-art methods.