mlsanityData to 14 Oct 2026

Towards High-Fidelity Text-Guided 3D Face Generation and Manipulation Using only Images

arXiv 2308.16758 ↗First posted 31 Aug 2023cs.CVcited by 12

Not on the map. Why?

Abstract (arXiv)

Generating 3D faces from textual descriptions has a multitude of applications, such as gaming, movie, and robotics. Recent progresses have demonstrated the success of unconditional 3D face generation and text-to-3D shape generation. However, due to the limited text-3D face data pairs, text-driven 3D face generation remains an open problem. In this paper, we propose a text-guided 3D faces generation method, refer as TG-3DFace, for generating realistic 3D faces using text guidance. Specifically, we adopt an unconditional 3D face generation framework and equip it with text conditions, which learns the text-guided 3D face generation with only text-2D face data. On top of that, we propose two text-to-face cross-modal alignment techniques, including the global contrastive learning and the fine-grained alignment module, to facilitate high semantic consistency between generated 3D faces and input texts. Besides, we present directional classifier guidance during the inference process, which encourages creativity for out-of-domain generations. Compared to the existing methods, TG-3DFace creates more realistic and aesthetically pleasing 3D faces, boosting 9% multi-view consistency (MVIC) over Latent3D. The rendered face images generated by TG-3DFace achieve higher FID and CLIP score than text-to-2D face/image generation models, demonstrating our superiority in generating realistic and semantic-consistent textures.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 49 of its 66 references that are in the map. First posted on arXiv 31 Aug 2023.

Builds on

49 of 66 references.

References · 49

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  5. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  6. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  8. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  9. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  10. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,849
  11. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  12. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,205
37 more
  1. 2016Generative Adversarial Text to Image Synthesiscited by 735
  2. 2022Autoregressive Image Generation using Residual Quantizationcited by 722
  3. 2020pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesiscited by 708
  4. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 694
  5. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  6. 2018BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentationcited by 645
  7. 2019MaskGAN: Towards Diverse and Interactive Facial Image Manipulationcited by 628
  8. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 577
  9. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  10. 2021Pivotal Tuning for Latent-based Editing of Real Imagescited by 427
  11. 2021Text2Mesh: Text-Driven Neural Stylization for Meshescited by 334
  12. 2019DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesiscited by 308
  13. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 307
  14. 2022CLIP-Mesh: Generating textured meshes from text using pretrained image-text modelscited by 281
  15. 2021CLIP-Forge: Towards Zero-Shot Text-to-Shape Generationcited by 254
  16. 2017StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 229
  17. 2019Controllable Text-to-Image Generationcited by 187
  18. 2018Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddingscited by 169
  19. 2022RigNeRF: Fully Controllable Neural 3D Portraitscited by 126
  20. 2021ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesiscited by 122
  21. 2021FENeRF: Face Editing in Neural Radiance Fieldscited by 112
  22. 2020AvatarMe: Realistically Renderable 3D Facial Reconstruction "in-the-wild"cited by 91
  23. 2017Semantic Image Synthesis via Adversarial Learningcited by 87
  24. 2022Towards Implicit Text-Guided 3D Shape Generationcited by 68
  25. 2021Improving Text-to-Image Synthesis Using Contrastive Learningcited by 62
  26. 2022ShapeCrafter: A Recursive Text-Conditioned 3D Shape Generation Modelcited by 62
  27. 2022ClipFace: Text-guided Editing of Textured 3D Morphable Modelscited by 60
  28. 2021AvatarMe++: Facial Shape and BRDF Inference with Photorealistic Rendering-Aware GANscited by 49
  29. 2021ClipMatrix: Text-controlled Creation of 3D Textured Meshescited by 40
  30. 2021M6-UFC: Unifying Multi-Modal Controls for Conditional Image Synthesis via Non-Autoregressive Generative Transformerscited by 36
  31. 2022Text and Image Guided 3D Avatar Generation and Manipulationcited by 35
  32. 2022AnyFace: Free-style Text-to-Face Synthesis and Manipulationcited by 33
  33. 20193DFaceGAN: Adversarial Nets for 3D Face Representation, Generation, and Translationcited by 27
  34. 2021Towards Open-World Text-Guided Face Image Generation and Manipulationcited by 25
  35. 2020Conditional Image Generation and Manipulation for User-Specified Contentcited by 20
  36. 2020Faces `a la Carte: Text-to-Face Generation via Attribute Disentanglementcited by 16
  37. 2019Text2FaceGAN: Face Generation from Fine Grained Textual Descriptionscited by 9

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 9; 2025: 1.

12 papers.

  1. 20243D Gaussian as a New Era: A Surveycited by 104
  2. 2024Deepfake Generation and Detection: A Benchmark and Surveycited by 67
  3. 2024ConsistentID: Portrait Generation with Multimodal Fine-Grained Identity Preservingcited by 43
  4. 2024DreamPhysics: Learning Physics-Based 3D Dynamics with Video Diffusion Priorscited by 41
  5. 2023DreamControl: Control-Based Text-to-3D Generation with 3D Self-Priorcited by 39
  6. 2023HeadArtist: Text-conditioned 3D Head Generation with Self Score Distillationcited by 22
  7. 2024InstructAvatar: Text-Guided Emotion and Motion Control for Avatar Generationcited by 16
  8. 2024HeadEvolver: Text to Head Avatars via Expressive and Attribute-Preserving Mesh Deformationcited by 11
  9. 2024Fast Text-to-3D-Aware Face Generation and Manipulation via Direct Cross-modal Mapping and Geometric Regularizationcited by 8
  10. 2024FaceGPT: Self-supervised Learning to Chat about 3D Human Facescited by 4
  11. 2024Revealing Directions for Text-guided 3D Face Editingcited by 2
  12. 2025Disentangling 3D from Large Vision-Language Models for Controlled Portrait Generationcited by 0