# High-Fidelity 3D Face Generation from Natural Language Descriptions

arXiv: https://arxiv.org/abs/2305.03302 · HTML: https://mlsanity.com/paper/2305.03302/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 60 references that are in the map.
- First posted on arXiv 5 May 2023.

## Builds on (most cited first, 30 of 45)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Generative Adversarial Text to Image Synthesis](https://mlsanity.com/paper/1605.05396/) (2016)
- [Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation](https://mlsanity.com/paper/2008.00951/) (2020)
- [AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks](https://mlsanity.com/paper/1711.10485/) (2017)
- [StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks](https://mlsanity.com/paper/1612.03242/) (2016)
- [Designing an Encoder for StyleGAN Image Manipulation](https://mlsanity.com/paper/2102.02766/) (2021)
- [Zero-Shot Text-Guided Object Generation with Dream Fields](https://mlsanity.com/paper/2112.01455/) (2021)
- [Learning an Animatable Detailed 3D Face Model from In-The-Wild Images](https://mlsanity.com/paper/2012.04012/) (2020)
- [Text2Mesh: Text-Driven Neural Stylization for Meshes](https://mlsanity.com/paper/2112.03221/) (2021)
- [Mixture of Volumetric Primitives for Efficient Neural Rendering](https://mlsanity.com/paper/2103.01954/) (2021)
- [3D Morphable Face Models -- Past, Present and Future](https://mlsanity.com/paper/1909.01815/) (2019)
- [I M Avatar: Implicit Morphable Head Avatars from Videos](https://mlsanity.com/paper/2112.07471/) (2021)
- [StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networks](https://mlsanity.com/paper/1710.10916/) (2017)
- [HeadNeRF: A Real-time NeRF-based Parametric Head Model](https://mlsanity.com/paper/2112.05637/) (2021)
- [Towards Fast, Accurate and Stable 3D Dense Face Alignment](https://mlsanity.com/paper/2009.09960/) (2020)
- [FaceScape: a Large-scale High Quality 3D Face Dataset and Detailed Riggable 3D Face Prediction](https://mlsanity.com/paper/2003.13989/) (2020)
- [Joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Network](https://mlsanity.com/paper/1803.07835/) (2018)
- [MirrorGAN: Learning Text-to-image Generation by Redescription](https://mlsanity.com/paper/1903.05854/) (2019)
- [Learning to Regress 3D Face Shape and Expression from an Image without 3D Supervision](https://mlsanity.com/paper/1905.06817/) (2019)
- [Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddings](https://mlsanity.com/paper/1803.08495/) (2018)
- [i3DMM: Deep Implicit 3D Morphable Model of Human Heads](https://mlsanity.com/paper/2011.14143/) (2020)
- [Learning What and Where to Draw](https://mlsanity.com/paper/1610.02454/) (2016)

## Built on by (most cited first, 30 of 33)

- [IDOL: Instant Photorealistic 3D Human Creation from a Single Image](https://mlsanity.com/paper/2412.14963/) (2024)
- [Text-Guided Generation and Editing of Compositional 3D Avatars](https://mlsanity.com/paper/2309.07125/) (2023)
- [AvatarBooth: High-Quality and Customizable 3D Human Avatar Generation](https://mlsanity.com/paper/2306.09864/) (2023)
- [FATE: Full-head Gaussian Avatar with Textural Editing from Monocular Video](https://mlsanity.com/paper/2411.15604/) (2024)
- [Anti-Aliased Neural Implicit Surfaces with Encoding Level of Detail](https://mlsanity.com/paper/2309.10336/) (2023)
- [HeadArtist: Text-conditioned 3D Head Generation with Self Score Distillation](https://mlsanity.com/paper/2312.07539/) (2023)
- [Mani-GS: Gaussian Splatting Manipulation with Triangular Mesh](https://mlsanity.com/paper/2405.17811/) (2024)
- [MeshUp: Multi-Target Mesh Deformation via Blended Score Distillation](https://mlsanity.com/paper/2408.14899/) (2024)
- [A Survey on 3D Human Avatar Modeling -- From Reconstruction to Generation](https://mlsanity.com/paper/2406.04253/) (2024)
- [AWOL: Analysis WithOut synthesis using Language](https://mlsanity.com/paper/2404.03042/) (2024)
- [Text-Guided 3D Face Synthesis -- From Generation to Editing](https://mlsanity.com/paper/2312.00375/) (2023)
- [Controllable 3D Face Generation with Conditional Style Code Diffusion](https://mlsanity.com/paper/2312.13941/) (2023)
- [Words in Motion: Extracting Interpretable Control Vectors for Motion Transformers](https://mlsanity.com/paper/2406.11624/) (2024)
- [Head360: Learning a Parametric 3D Full-Head for Free-View Synthesis in 360deg](https://mlsanity.com/paper/2408.00296/) (2024)
- [Fast Text-to-3D-Aware Face Generation and Manipulation via Direct Cross-modal Mapping and Geometric Regularization](https://mlsanity.com/paper/2403.06702/) (2024)
- [MMoFusion: Multi-modal Co-Speech Motion Generation with Diffusion Model](https://mlsanity.com/paper/2403.02905/) (2024)
- [Towards Native Generative Model for 3D Head Avatar](https://mlsanity.com/paper/2410.01226/) (2024)
- [Efficient Text-Guided 3D-Aware Portrait Generation with Score Distillation Sampling on Distribution](https://mlsanity.com/paper/2306.02083/) (2023)
- [FaceGPT: Self-supervised Learning to Chat about 3D Human Faces](https://mlsanity.com/paper/2406.07163/) (2024)
- [StrandHead: Text to Hair-Disentangled 3D Head Avatars Using Human-Centric Priors](https://mlsanity.com/paper/2412.11586/) (2024)
- [GenCA: A Text-conditioned Generative Model for Realistic and Drivable Codec Avatars](https://mlsanity.com/paper/2408.13674/) (2024)
- [Evaluating Deepfake Detectors in the Wild](https://mlsanity.com/paper/2507.21905/) (2025)
- [Spherical Geometry Diffusion: Generating High-quality 3D Face Geometry via Sphere-anchored Representations](https://mlsanity.com/paper/2601.13371/) (2026)
- [EmoteGPT: 3D Human Facial Expressions from Natural Language Descriptions](https://mlsanity.com/paper/2607.02674/) (2026)
- [Reality's Canvas, Language's Brush: Crafting 3D Avatars from Monocular Video](https://mlsanity.com/paper/2312.04784/) (2023)
- [Multimodal Conditional 3D Face Geometry Generation](https://mlsanity.com/paper/2407.01074/) (2024)
- [Bringing Diversity from Diffusion Models to Semantic-Guided Face Asset Generation](https://mlsanity.com/paper/2504.15259/) (2025)
- [Disentangling 3D from Large Vision-Language Models for Controlled Portrait Generation](https://mlsanity.com/paper/2506.14015/) (2025)
- [Imagining Alternatives: Towards High-Resolution 3D Counterfactual Medical Image Generation via Language Guidance](https://mlsanity.com/paper/2509.05978/) (2025)
- [Dual Diffusion Models for Multi-modal Guided 3D Avatar Generation](https://mlsanity.com/paper/2603.04307/) (2026)

## Abstract (arXiv)

Synthesizing high-quality 3D face models from natural language descriptions is very valuable for many applications, including avatar creation, virtual reality, and telepresence. However, little research ever tapped into this task. We argue the major obstacle lies in 1) the lack of high-quality 3D face data with descriptive text annotation, and 2) the complex mapping relationship between descriptive language space and shape/appearance space. To solve these problems, we build Describe3D dataset, the first large-scale dataset with fine-grained text descriptions for text-to-3D face generation task. Then we propose a two-stage framework to first generate a 3D face that matches the concrete descriptions, then optimize the parameters in the 3D shape and texture space with abstract description to refine the 3D face model. Extensive experimental results show that our method can produce a faithful 3D face that conforms to the input descriptions with higher accuracy and quality than previous methods. The code and Describe3D dataset are released at https://github.com/zhuhao-nju/describe3d .