# Instant 3D Human Avatar Generation using Image Diffusion Models

arXiv: https://arxiv.org/abs/2406.07516 · HTML: https://mlsanity.com/paper/2406.07516/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models.
- Builds on 71 of its 82 references that are in the map.
- First posted on arXiv 11 Jun 2024.

## Placed in

- Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/)

## Builds on (most cited first, 30 of 71)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022)
- [Zero-1-to-3: Zero-shot One Image to 3D Object](https://mlsanity.com/paper/2303.11328/) (2023)
- [Magic3D: High-Resolution Text-to-3D Content Creation](https://mlsanity.com/paper/2211.10440/) (2022)
- [DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation](https://mlsanity.com/paper/2309.16653/) (2023)
- [PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization](https://mlsanity.com/paper/1905.05172/) (2019)
- [PaLI: A Jointly-Scaled Multilingual Language-Image Model](https://mlsanity.com/paper/2209.06794/) (2022)
- [Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loop](https://mlsanity.com/paper/1909.12828/) (2019)
- [One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization](https://mlsanity.com/paper/2306.16928/) (2023)
- [PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization](https://mlsanity.com/paper/2004.00452/) (2020)
- [VideoPoet: A Large Language Model for Zero-Shot Video Generation](https://mlsanity.com/paper/2312.14125/) (2023)
- [Phenaki: Variable Length Video Generation From Open Domain Textual Description](https://mlsanity.com/paper/2210.02399/) (2022)
- [Lumiere: A Space-Time Diffusion Model for Video Generation](https://mlsanity.com/paper/2401.12945/) (2024)
- [VITON: An Image-based Virtual Try-on Network](https://mlsanity.com/paper/1711.08447/) (2017)
- [ICON: Implicit Clothed humans Obtained from Normals](https://mlsanity.com/paper/2112.09127/) (2021)
- [ARCH: Animatable Reconstruction of Clothed Humans](https://mlsanity.com/paper/2004.04572/) (2020)
- [Video Based Reconstruction of 3D People Models](https://mlsanity.com/paper/1803.04758/) (2018)

## Built on by (most cited first, 14 of 14)

- [PERSONA: Personalized Whole-Body 3D Avatar with Pose-Driven Deformations from a Single Image](https://mlsanity.com/paper/2508.09973/) (2025)
- [LIFe-GoM: Generalizable Human Rendering with Learned Iterative Feedback Over Multi-Resolution Gaussians-on-Mesh](https://mlsanity.com/paper/2502.09617/) (2025)
- [AdaHuman: Animatable Detailed 3D Human Generation with Compositional Multiview Diffusion](https://mlsanity.com/paper/2505.24877/) (2025)
- [Disentangled Clothed Avatar Generation with Layered Representation](https://mlsanity.com/paper/2501.04631/) (2025)
- [StdGEN: Semantic-Decomposed 3D Character Generation from Single Images](https://mlsanity.com/paper/2411.05738/) (2024)
- [Generative Human Geometry Distribution](https://mlsanity.com/paper/2503.01448/) (2025)
- [DeClotH: Decomposable 3D Cloth and Human Body Reconstruction from a Single Image](https://mlsanity.com/paper/2503.19373/) (2025)
- [Dream, Lift, Animate: From Single Images to Animatable Gaussian Avatars](https://mlsanity.com/paper/2507.15979/) (2025)
- [PARTE: Part-Guided Texturing for 3D Human Reconstruction from a Single Image](https://mlsanity.com/paper/2507.17332/) (2025)
- [HuGeDiff: 3D Human Generation via Diffusion with Gaussian Splatting](https://mlsanity.com/paper/2506.04351/) (2025)
- [Realistic Clothed Human and Object Joint Reconstruction from a Single Image](https://mlsanity.com/paper/2502.18150/) (2025)
- [NoiseShift: Resolution-Aware Noise Recalibration for Better Low-Resolution Image Generation](https://mlsanity.com/paper/2510.02307/) (2025)
- [NoPo-Avatar: Generalizable and Animatable Avatars from Sparse Inputs without Human Poses](https://mlsanity.com/paper/2511.16673/) (2025)
- [AvatarDynamizer: From Static to Dynamic Human Avatars via Generative Dynamic Textures](https://mlsanity.com/paper/2608.19900/) (2026)

## Abstract (arXiv)

We present AvatarPopUp, a method for fast, high quality 3D human avatar generation from different input modalities, such as images and text prompts and with control over the generated pose and shape. The common theme is the use of diffusion-based image generation networks that are specialized for each particular task, followed by a 3D lifting network. We purposefully decouple the generation from the 3D modeling which allow us to leverage powerful image synthesis priors, trained on billions of text-image pairs. We fine-tune latent diffusion networks with additional image conditioning for image generation and back-view prediction, and to support qualitatively different multiple 3D hypotheses. Our partial fine-tuning approach allows to adapt the networks for each task without inducing catastrophic forgetting. In our experiments, we demonstrate that our method produces accurate, high-quality 3D avatars with diverse appearance that respect the multimodal text, image, and body control signals. Our approach can produce a 3D model in as few as 2 seconds, a four orders of magnitude speedup wrt the vast majority of existing methods, most of which solve only a subset of our tasks, and with fewer controls. AvatarPopUp enables applications that require the controlled 3D generation of human avatars at scale. The project website can be found at https://www.nikoskolot.com/avatarpopup/.