mlsanityData to 7 Oct 2026

3D-aware Image Generation and Editing with Multi-modal Conditions

arXiv 2403.06470 ↗First posted 11 Mar 2024cs.CVcited by 2

Not on the map. Why?

Abstract (arXiv)

3D-consistent image generation from a single 2D semantic label is an important and challenging research topic in computer graphics and computer vision. Although some related works have made great progress in this field, most of the existing methods suffer from poor disentanglement performance of shape and appearance, and lack multi-modal control. In this paper, we propose a novel end-to-end 3D-aware image generation and editing model incorporating multiple types of conditional inputs, including pure noise, text and reference image. On the one hand, we dive into the latent space of 3D Generative Adversarial Networks (GANs) and propose a novel disentanglement strategy to separate appearance features from shape features during the generation process. On the other hand, we propose a unified framework for flexible image generation and editing tasks with multi-modal conditions. Our method can generate diverse images with distinct noises, edit the attribute through a text description and conduct style transfer by giving a reference RGB image. Extensive experiments demonstrate that the proposed method outperforms alternative approaches both qualitatively and quantitatively on image generation and editing.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 45 of its 56 references that are in the map. First posted on arXiv 11 Mar 2024.

Builds on

45 of 56 references.

References · 45

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  3. 2022Generative Adversarial Networkscited by 5,603
  4. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  5. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  6. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  7. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
  8. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,295
  9. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,204
  10. 2021Alias-Free Generative Adversarial Networkscited by 1,181
  11. 2019StarGAN v2: Diverse Image Synthesis for Multiple Domainscited by 1,095
  12. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
33 more
  1. 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 784
  2. 2020pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesiscited by 708
  3. 2020GRAF: Generative Radiance Fields for 3D-Aware Image Synthesiscited by 688
  4. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  5. 2018BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentationcited by 645
  6. 2019MaskGAN: Towards Diverse and Interactive Facial Image Manipulationcited by 624
  7. 2020GANSpace: Discovering Interpretable GAN Controlscited by 612
  8. 2022Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generationcited by 597
  9. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 575
  10. 2021StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesiscited by 477
  11. 2020Closed-Form Factorization of Latent Semantics in GANscited by 407
  12. 2020InterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANscited by 393
  13. 2020StyleFlow: Attribute-conditioned Exploration of StyleGAN-Generated Images using Conditional Continuous Normalizing Flowscited by 348
  14. 2021Deep ViT Features as Dense Visual Descriptorscited by 327
  15. 2021StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generationcited by 289
  16. 2019SEAN: Image Synthesis with Semantic Region-Adaptive Normalizationcited by 253
  17. 2019GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstructioncited by 150
  18. 2023Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generationcited by 133
  19. 2020HOPE-Net: A Graph-based Model for Hand-Object Pose Estimationcited by 112
  20. 2021FENeRF: Face Editing in Neural Radiance Fieldscited by 112
  21. 2021Talk-to-Edit: Fine-Grained Facial Editing via Dialogcited by 102
  22. 2020AvatarMe: Realistically Renderable 3D Facial Reconstruction "in-the-wild"cited by 91
  23. 2016Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Unitscited by 86
  24. 2020SofGAN: A Portrait Image Generator with Dynamic Stylingcited by 65
  25. 2017Unrestricted Facial Geometry Reconstruction Using Image-to-Image Translationcited by 56
  26. 2022Multi-View Consistent Generative Adversarial Networks for 3D-aware Image Synthesiscited by 43
  27. 2022Diffusion Guided Domain Adaptation of Image Generatorscited by 35
  28. 2019Synthesizing Coupled 3D Face Modalities by Trunk-Branch Generative Adversarial Networkscited by 34
  29. 20213D-Aware Semantic-Guided Generative Model for Human Synthesiscited by 31
  30. 2018Semi-supervised Adversarial Learning to Generate Photorealistic Face Images of New Identities from 3D Morphable Modelcited by 28
  31. 20233D-aware Conditional Image Synthesiscited by 24
  32. 2023Learning 3D-aware Image Synthesis with Unknown Pose Distributioncited by 14
  33. 2023Efficient Text-Guided 3D-Aware Portrait Generation with Score Distillation Sampling on Distributioncited by 4

Built on by

Cited by year

Citing papers per year: 2024: 2.

2 papers.

  1. 2024Revealing Directions for Text-guided 3D Face Editingcited by 2
  2. 2024Pomo3D: 3D-Aware Portrait Accessorizing and Morecited by 0