# AvatarFusion: Zero-shot Generation of Clothing-Decoupled 3D Avatars Using 2D Diffusion

arXiv: https://arxiv.org/abs/2307.06526 · HTML: https://mlsanity.com/paper/2307.06526/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 58 references that are in the map.
- First posted on arXiv 13 Jul 2023.

## Builds on (most cited first, 30 of 43)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction](https://mlsanity.com/paper/2106.10689/) (2021)
- [D-NeRF: Neural Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2011.13961/) (2020)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [Magic3D: High-Resolution Text-to-3D Content Creation](https://mlsanity.com/paper/2211.10440/) (2022)
- [Efficient Geometry-aware 3D Generative Adversarial Networks](https://mlsanity.com/paper/2112.07945/) (2021)
- [Nerfies: Deformable Neural Radiance Fields](https://mlsanity.com/paper/2011.12948/) (2020)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [Diffusion Probabilistic Models for 3D Point Cloud Generation](https://mlsanity.com/paper/2103.01458/) (2021)
- [Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision](https://mlsanity.com/paper/1912.07372/) (2019)
- [Point-E: A System for Generating 3D Point Clouds from Complex Prompts](https://mlsanity.com/paper/2212.08751/) (2022)
- [ClipCap: CLIP Prefix for Image Captioning](https://mlsanity.com/paper/2111.09734/) (2021)
- [3D Shape Generation and Completion through Point-Voxel Diffusion](https://mlsanity.com/paper/2104.03670/) (2021)
- [Zero-Shot Text-Guided Object Generation with Dream Fields](https://mlsanity.com/paper/2112.01455/) (2021)
- [Latent-NeRF for Shape-Guided Generation of 3D Shapes and Textures](https://mlsanity.com/paper/2211.07600/) (2022)
- [Animatable Neural Radiance Fields for Modeling Dynamic Human Bodies](https://mlsanity.com/paper/2105.02872/) (2021)
- [CRIS: CLIP-Driven Referring Image Segmentation](https://mlsanity.com/paper/2111.15174/) (2021)
- [Text2Mesh: Text-Driven Neural Stylization for Meshes](https://mlsanity.com/paper/2112.03221/) (2021)
- [Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion](https://mlsanity.com/paper/2212.06135/) (2022)
- [TEXTure: Text-Guided Texturing of 3D Shapes](https://mlsanity.com/paper/2302.01721/) (2023)
- [StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generation](https://mlsanity.com/paper/2112.11427/) (2021)
- [Decoupling Zero-Shot Semantic Segmentation](https://mlsanity.com/paper/2112.07910/) (2021)
- [Segment and Track Anything](https://mlsanity.com/paper/2305.06558/) (2023)
- [Neural Importance Sampling](https://mlsanity.com/paper/1808.03856/) (2018)
- [SMPLicit: Topology-aware Generative Model for Clothed People](https://mlsanity.com/paper/2103.06871/) (2021)

## Built on by (most cited first, 20 of 20)

- [MIGC: Multi-Instance Generation Controller for Text-to-Image Synthesis](https://mlsanity.com/paper/2402.05408/) (2024)
- [SIFU: Side-view Conditioned Implicit Function for Real-world Usable Clothed Human Reconstruction](https://mlsanity.com/paper/2312.06704/) (2023)
- [MIGC++: Advanced Multi-Instance Generation Controller for Image Synthesis](https://mlsanity.com/paper/2407.02329/) (2024)
- [Disentangled Clothed Avatar Generation from Text Descriptions](https://mlsanity.com/paper/2312.05295/) (2023)
- [SEEAvatar: Photorealistic Text-to-3D Avatar Generation with Constrained Geometry and Appearance](https://mlsanity.com/paper/2312.08889/) (2023)
- [HeadArtist: Text-conditioned 3D Head Generation with Self Score Distillation](https://mlsanity.com/paper/2312.07539/) (2023)
- [CapHuman: Capture Your Moments in Parallel Universes](https://mlsanity.com/paper/2402.00627/) (2024)
- [GarmentDreamer: 3DGS Guided Garment Synthesis with Diverse Geometry and Texture Details](https://mlsanity.com/paper/2405.12420/) (2024)
- [Enhancing High-Resolution 3D Generation through Pixel-wise Gradient Clipping](https://mlsanity.com/paper/2310.12474/) (2023)
- [STAR: Skeleton-aware Text-based 4D Avatar Generation with In-Network Motion Retargeting](https://mlsanity.com/paper/2406.04629/) (2024)
- [Controllable 3D Face Generation with Conditional Style Code Diffusion](https://mlsanity.com/paper/2312.13941/) (2023)
- [ClotheDreamer: Text-Guided Garment Generation with 3D Gaussians](https://mlsanity.com/paper/2406.16815/) (2024)
- [X-Dreamer: Creating High-quality 3D Content by Bridging the Domain Gap Between Text-to-2D and Text-to-3D Generation](https://mlsanity.com/paper/2312.00085/) (2023)
- [UVMap-ID: A Controllable and Personalized UV Map Generative Model](https://mlsanity.com/paper/2404.14568/) (2024)
- [HumanCoser: Layered 3D Human Generation via Semantic-Aware Diffusion Model](https://mlsanity.com/paper/2408.11357/) (2024)
- [PlacidDreamer: Advancing Harmony in Text-to-3D Generation](https://mlsanity.com/paper/2407.13976/) (2024)
- [StdGEN: Semantic-Decomposed 3D Character Generation from Single Images](https://mlsanity.com/paper/2411.05738/) (2024)
- [DreamBarbie: Text to Barbie-Style 3D Avatars](https://mlsanity.com/paper/2408.09126/) (2024)
- [A Text-to-3D Framework for Joint Generation of CG-Ready Humans and Compatible Garments](https://mlsanity.com/paper/2503.12052/) (2025)
- [StdGEN++: A Comprehensive System for Semantic-Decomposed 3D Character Generation](https://mlsanity.com/paper/2601.07660/) (2026)

## Abstract (arXiv)

Large-scale pre-trained vision-language models allow for the zero-shot text-based generation of 3D avatars. The previous state-of-the-art method utilized CLIP to supervise neural implicit models that reconstructed a human body mesh. However, this approach has two limitations. Firstly, the lack of avatar-specific models can cause facial distortion and unrealistic clothing in the generated avatars. Secondly, CLIP only provides optimization direction for the overall appearance, resulting in less impressive results. To address these limitations, we propose AvatarFusion, the first framework to use a latent diffusion model to provide pixel-level guidance for generating human-realistic avatars while simultaneously segmenting clothing from the avatar's body. AvatarFusion includes the first clothing-decoupled neural implicit avatar model that employs a novel Dual Volume Rendering strategy to render the decoupled skin and clothing sub-models in one space. We also introduce a novel optimization method, called Pixel-Semantics Difference-Sampling (PS-DS), which semantically separates the generation of body and clothes, and generates a variety of clothing styles. Moreover, we establish the first benchmark for zero-shot text-to-avatar generation. Our experimental results demonstrate that our framework outperforms previous approaches, with significant improvements observed in all metrics. Additionally, since our model is clothing-decoupled, we can exchange the clothes of avatars. Code are available on our project page https://hansenhuang0823.github.io/AvatarFusion.