mlsanityData to 7 Oct 2026

Multimodal Conditional 3D Face Geometry Generation

arXiv 2407.01074 ↗First posted 1 Jul 2024cs.CV cs.GRcited by 0

Not on the map. Why?

Abstract (arXiv)

We present a new method for multimodal conditional 3D face geometry generation that allows user-friendly control over the output identity and expression via a number of different conditioning signals. Within a single model, we demonstrate 3D faces generated from artistic sketches, portrait photos, Canny edges, FLAME face model parameters, 2D face landmarks, or text prompts. Our approach is based on a diffusion process that generates 3D geometry in a 2D parameterized UV domain. Geometry generation passes each conditioning signal through a set of cross-attention layers (IP-Adapter), one set for each user-defined conditioning signal. The result is an easy-to-use 3D face generation tool that produces topology-consistent, high-quality geometry with fine-grain user control.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 39 of its 52 references that are in the map. First posted on arXiv 1 Jul 2024.

Builds on

39 of 52 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  7. 2020Denoising Diffusion Implicit Modelscited by 9,655
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  9. 2022Classifier-Free Diffusion Guidancecited by 5,717
  10. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  11. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  12. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
27 more
  1. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  2. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  3. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  4. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  5. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,439
  6. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,419
  7. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  8. 2021Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesiscited by 459
  9. 2021Diffusion Autoencoders: Toward a Meaningful and Decodable Representationcited by 447
  10. 2022Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusioncited by 309
  11. 2018Joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Networkcited by 195
  12. 2022Sketch-Guided Text-to-Image Diffusion Modelscited by 180
  13. 2023Generalization in diffusion models arises from geometry-adaptive harmonic representationscited by 179
  14. 2023HumanNorm: Learning Normal Diffusion Model for High-quality and Realistic 3D Human Generationcited by 88
  15. 2023DreamFace: Progressive Generation of Animatable 3D Faces under Text Guidancecited by 79
  16. 2023DiffusionRig: Learning Personalized Priors for Facial Appearance Editingcited by 74
  17. 2023HeadSculpt: Crafting 3D Head Avatars with Textcited by 48
  18. 2023DiffusionAvatars: Deferred Diffusion for High-fidelity 3D Head Avatarscited by 42
  19. 2023High-Fidelity 3D Face Generation from Natural Language Descriptionscited by 32
  20. 2024It's All About Your Sketch: Democratising Sketch Control in Diffusion Modelscited by 28
  21. 2023Control3Diff: Learning Controllable 3D Diffusion Models from Single-view Imagescited by 27
  22. 2023HeadArtist: Text-conditioned 3D Head Generation with Self Score Distillationcited by 20
  23. 20234D Facial Expression Diffusion Modelcited by 15
  24. 2024UltrAvatar: A Realistic Animatable 3D Avatar Diffusion Model with Authenticity Guided Texturescited by 13
  25. 2023Text-Guided 3D Face Synthesis -- From Generation to Editingcited by 8
  26. 2023Articulated 3D Head Avatar Generation using Text-to-Image Diffusion Modelscited by 6
  27. 2024ShapeFusion: A 3D diffusion model for localized shape editingcited by 6

Built on by

Not cited yet.