mlsanityData to 7 Oct 2026

Text and Image Guided 3D Avatar Generation and Manipulation

arXiv 2202.06079 ↗First posted 12 Feb 2022cs.CV cs.GR cs.LGcited by 35

Not on the map. Why?

Abstract (arXiv)

The manipulation of latent space has recently become an interesting topic in the field of generative models. Recent research shows that latent directions can be used to manipulate images towards certain attributes. However, controlling the generation process of 3D generative models remains a challenge. In this work, we propose a novel 3D manipulation method that can manipulate both the shape and texture of the model using text or image-based prompts such as 'a young face' or 'a surprised face'. We leverage the power of Contrastive Language-Image Pre-training (CLIP) model and a pre-trained 3D GAN model designed to generate face avatars, and create a fully differentiable rendering pipeline to manipulate meshes. More specifically, our method takes an input latent code and modifies it such that the target attribute specified by a text or image prompt is present or enhanced, while leaving other attributes largely unaffected. Our method requires only 5 minutes per manipulation, and we demonstrate the effectiveness of our approach with extensive results and comparisons.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Builds on 48 of its 60 references that are in the map. First posted on arXiv 12 Feb 2022.

Builds on

48 of 60 references.

References · 48

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  3. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  4. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  5. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  6. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  7. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  8. 2017Improved Training of Wasserstein GANscited by 2,223
  9. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
  10. 2018Learning Implicit Fields for Generative Shape Modelingcited by 1,060
  11. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  12. 2016A Point Set Generation Network for 3D Object Reconstruction from a Single Imagecited by 972
36 more
  1. 2018PointRCNN: 3D Object Proposal Generation and Detection from Point Cloudcited by 939
  2. 2020Accelerating 3D Deep Learning with PyTorch3Dcited by 781
  3. 2020GANSpace: Discovering Interpretable GAN Controlscited by 612
  4. 20163D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstructioncited by 587
  5. 2018AtlasNet: A Papier-Mach'e Approach to Learning 3D Surface Generationcited by 539
  6. 2020Closed-Form Factorization of Latent Semantics in GANscited by 407
  7. 2020InterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANscited by 393
  8. 2021CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fieldscited by 369
  9. 2021Text2Mesh: Text-Driven Neural Stylization for Meshescited by 332
  10. 2020Unsupervised Discovery of Interpretable Directions in the GAN Latent Spacecited by 288
  11. 2021CLIP-Forge: Towards Zero-Shot Text-to-Shape Generationcited by 254
  12. 2019On the "steerability" of generative adversarial networkscited by 247
  13. 2019GANalyze: Toward Visual Definitions of Cognitive Image Propertiescited by 193
  14. 20193D Point Cloud Generative Adversarial Network Based on Tree Structured Graph Convolutionscited by 173
  15. 2019GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstructioncited by 150
  16. 2017Learning Efficient Point Cloud Generation for Dense 3D Object Reconstructioncited by 133
  17. 2016Regressing Robust and Discriminative 3D Morphable Models with a very Deep Neural Networkcited by 109
  18. 2017Self-supervised Multi-level Face Model Learning for Monocular Reconstruction at over 250 Hzcited by 91
  19. 2021H3D-Net: Few-Shot High-Fidelity 3D Head Reconstructioncited by 81
  20. 2018FML: Face Model Learning from Videoscited by 78
  21. 2018MeshCNN: A Network with an Edgecited by 69
  22. 2017UV-GAN: Adversarial Facial UV Map Completion for Pose-invariant Face Recognitioncited by 62
  23. 2014Category-Specific Object Reconstruction from a Single Imagecited by 56
  24. 2017Unrestricted Facial Geometry Reconstruction Using Image-to-Image Translationcited by 56
  25. 2018ArcFace: Additive Angular Margin Loss for Deep Face Recognitioncited by 56
  26. 2021LatentCLR: A Contrastive Learning Approach for Unsupervised Discovery of Interpretable Directionscited by 52
  27. 2021Sketch2Mesh: Reconstructing and Editing 3D Shapes from Sketchescited by 50
  28. 2020Progressive Point Cloud Deconvolution Generation Networkcited by 46
  29. 20173D Face Morphable Models "In-the-Wild"cited by 44
  30. 2021StyleMC: Multi-Channel Based Fast Text-Guided Image Generation and Manipulationcited by 40
  31. 2019Synthesizing Coupled 3D Face Modalities by Trunk-Branch Generative Adversarial Networkscited by 34
  32. 20173D Face Reconstruction with Geometry Details from a Single Imagecited by 27
  33. 2017Pose-Invariant Face Alignment with a Single CNNcited by 11
  34. 2015Pose-Invariant 3D Face Alignmentcited by 8
  35. 2021Intuitive Shape Editing in Latent Spacecited by 7
  36. 2018A Self-Supervised Bootstrap Method for Single-Image 3D Face Reconstructioncited by 2

Built on by

Cited by year

Citing papers per year: 2022: 7; 2023: 12; 2024: 12; 2025: 4.

35 papers.

Sort:

By area:

  1. 2023One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimizationcited by 555
  2. 2023OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understandingcited by 183
  3. 2022PartSLIP: Low-Shot Part Segmentation for 3D Point Clouds via Pretrained Image-Language Modelscited by 117
  4. 20243D Gaussian as a New Era: A Surveycited by 102
  5. 2022ClipFace: Text-guided Editing of Textured 3D Morphable Modelscited by 59
  6. 2022CLIP-Actor: Text-Driven Recommendation and Stylization for Animating Human Meshescited by 41
  7. 2023Text-Guided Generation and Editing of Compositional 3D Avatarscited by 39
  8. 2023High-Fidelity 3D Face Generation from Natural Language Descriptionscited by 32
  9. 2023AI-Generated Content (AIGC) for Various Data Modalities: A Surveycited by 25
  10. 2022Text2Light: Zero-Shot Text-Driven HDR Panorama Generationcited by 24
  11. 2023Local 3D Editing via 3D Distillation of CLIP Knowledgecited by 20
  12. 2022CLIP-PAE: Projection-Augmentation Embedding to Extract Relevant Features for a Disentangled, Interpretable, and Controllable Text-Guided Face Manipulationcited by 19
23 more
  1. 2023Sculpting Holistic 3D Representation in Contrastive Language-Image-3D Pre-trainingcited by 17
  2. 2024InstructAvatar: Text-Guided Emotion and Motion Control for Avatar Generationcited by 16
  3. 2023Zero-Shot Text-to-Parameter Translation for Game Character Auto-Creationcited by 15
  4. 2022Volumetric Disentanglement for 3D Scene Manipulationcited by 13
  5. 2024UltrAvatar: A Realistic Animatable 3D Avatar Diffusion Model with Authenticity Guided Texturescited by 13
  6. 2023Towards High-Fidelity Text-Guided 3D Face Generation and Manipulation Using only Imagescited by 12
  7. 2024Portrait Video Editing Empowered by Multimodal Generative Priorscited by 12
  8. 20223D-TOGO: Towards Text-Guided Cross-Category 3D Object Generationcited by 8
  9. 2024Fast Text-to-3D-Aware Face Generation and Manipulation via Direct Cross-modal Mapping and Geometric Regularizationcited by 8
  10. 2023Articulated 3D Head Avatar Generation using Text-to-Image Diffusion Modelscited by 6
  11. 2024Towards High-Quality 3D Motion Transfer with Realistic Apparel Animationcited by 5
  12. 2024Magic3DSketch: Create Colorful 3D Models From Sketch-Based 3D Modeling Guided by Text and Language-Image Pre-Trainingcited by 5
  13. 2025Unified Cross-Scale 3D Generation and Understanding via Autoregressive Modelingcited by 5
  14. 20243DStyleGLIP: Part-Tailored Text-Guided 3D Neural Stylizationcited by 4
  15. 2024A Comprehensive Taxonomy and Analysis of Talking Head Synthesis: Techniques for Portrait Generation, Driving Mechanisms, and Editingcited by 4
  16. 2023Intelligent Generation of Graphical Game Assets: A Conceptual Framework and Systematic Review of the State of the Artcited by 1
  17. 2024MeshBrush: Painting the Anatomical Mesh with Neural Stylization for Endoscopycited by 1
  18. 2024MM-Mixing: Multi-Modal Mixing Alignment for 3D Understandingcited by 1
  19. 2025Follow Your Motion: A Generic Temporal Consistency Portrait Editing Framework with Trajectory Guidancecited by 1
  20. 2023CosAvatar: Consistent and Animatable Portrait Video Tuning with Text Promptcited by 0
  21. 2024Enhancing Texture Generation with High-Fidelity Using Advanced Texture Priorscited by 0
  22. 2025Disentangling 3D from Large Vision-Language Models for Controlled Portrait Generationcited by 0
  23. 2025SIE3D: Single-Image Expressive 3D Avatar Generation via Semantic Embedding and Perceptual Expression Losscited by 0