mlsanityData to 7 Oct 2026

AvatarMMC: 3D Head Avatar Generation and Editing with Multi-Modal Conditioning

arXiv 2402.05803 ↗First posted 8 Feb 2024cs.CV cs.GRcited by 0

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

We introduce an approach for 3D head avatar generation and editing with multi-modal conditioning based on a 3D Generative Adversarial Network (GAN) and a Latent Diffusion Model (LDM). 3D GANs can generate high-quality head avatars given a single or no condition. However, it is challenging to generate samples that adhere to multiple conditions of different modalities. On the other hand, LDMs excel at learning complex conditional distributions. To this end, we propose to exploit the conditioning capabilities of LDMs to enable multi-modal control over the latent space of a pre-trained 3D GAN. Our method can generate and edit 3D head avatars given a mixture of control signals such as RGB input, segmentation masks, and global attributes. This provides better control over the generation and editing of synthetic avatars both globally and locally. Experiments show that our proposed approach outperforms a solely GAN-based approach both qualitatively and quantitatively on generation and editing tasks. To the best of our knowledge, our approach is the first to introduce multi-modal conditioning to 3D avatar generation and editing. hrefavatarmmc-sig24.github.ioProject Page

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 51 of its 78 references that are in the map. First posted on arXiv 8 Feb 2024.

Builds on

51 of 78 references.

References · 51

  1. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  8. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  9. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  10. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  11. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  12. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
38 more
  1. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  2. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,295
  3. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  4. 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 784
  5. 2023K-Planes: Explicit Radiance Fields in Space, Time, and Appearancecited by 742
  6. 2016Generative Adversarial Text to Image Synthesiscited by 735
  7. 2023MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editingcited by 722
  8. 2019Interpreting the Latent Space of GANs for Semantic Face Editingcited by 691
  9. 2020GRAF: Generative Radiance Fields for 3D-Aware Image Synthesiscited by 688
  10. 2019Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?cited by 675
  11. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  12. 2019MaskGAN: Towards Diverse and Interactive Facial Image Manipulationcited by 624
  13. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  14. 2021StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesiscited by 477
  15. 2021Pivotal Tuning for Latent-based Editing of Real Imagescited by 427
  16. 2019Image2StyleGAN++: How to Edit the Embedded Images?cited by 356
  17. 2020StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generationcited by 331
  18. 2021StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generationcited by 289
  19. 2021GRAM: Generative Radiance Manifolds for 3D-Aware Image Generationcited by 216
  20. 2021More Control for Free! Image Synthesis with Semantic Diffusion Guidancecited by 211
  21. 2020BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Imagescited by 166
  22. 2019ManiGAN: Text-Guided Image Manipulationcited by 156
  23. 2021CIPS-3D: A 3D-Aware Generator of GANs Based on Conditionally-Independent Pixel Synthesiscited by 149
  24. 2022VoxGRAF: Fast 3D-Aware Image Synthesis with Sparse Voxel Gridscited by 133
  25. 2022Next3D: Generative Neural Texture Rasterization for 3D-Aware Head Avatarscited by 124
  26. 2021ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesiscited by 122
  27. 20163D Shape Induction from 2D Views of Multiple Objectscited by 121
  28. 2023Collaborative Diffusion for Multi-Modal Face Generation and Editingcited by 116
  29. 2022EpiGRAF: Rethinking training of 3D GANscited by 108
  30. 2022GIRAFFE HD: A High-Resolution 3D-aware Generative Modelcited by 86
  31. 2018Talking Face Generation by Conditional Recurrent Adversarial Networkcited by 84
  32. 2022Pix2NeRF: Unsupervised Conditional π-GAN for Single Image to Neural Radiance Fields Translationcited by 73
  33. 2022gDNA: Towards Generative Detailed Neural Avatarscited by 72
  34. 2021Controllable and Compositional Generation with Latent-Space Energy-Based Modelscited by 64
  35. 2023Real-Time Radiance Fields for Single-Image Portrait View Synthesiscited by 64
  36. 2022Generative Multiplane Images: Making a 2D GAN 3D-Awarecited by 57
  37. 2021Generative Occupancy Fields for 3D Surface-Aware Image Synthesiscited by 51
  38. 2022Multi-View Consistent Generative Adversarial Networks for 3D-aware Image Synthesiscited by 43

50 most cited of 51 All 51 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion
  2. 2023HeadSculpt: Crafting 3D Head Avatars with Text
  3. 2024Instant 3D Human Avatar Generation using Image Diffusion Models
  4. 2023AvatarStudio: Text-driven Editing of 3D Dynamic Human Head Avatars
  5. 2024RodinHD: High-Fidelity 3D Avatar Generation with Diffusion Models
  6. 2023StyleAvatar3D: Leveraging Image-Text Diffusion Models for High-Fidelity 3D Avatar Generation
  7. 2024GenCA: A Text-conditioned Generative Model for Realistic and Drivable Codec Avatars
  8. 2024DynamicAvatars: Accurate Dynamic Facial Avatars Reconstruction and Precise Editing with Diffusion Models