# AvatarGen: A 3D Generative Model for Animatable Human Avatars

arXiv: https://arxiv.org/abs/2211.14589 · HTML: https://mlsanity.com/paper/2211.14589/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- Builds on 54 of its 75 references that are in the map.
- First posted on arXiv 26 Nov 2022.

## Builds on (most cited first, 30 of 54)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction](https://mlsanity.com/paper/2106.10689/) (2021)
- [D-NeRF: Neural Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2011.13961/) (2020)
- [Efficient Geometry-aware 3D Generative Adversarial Networks](https://mlsanity.com/paper/2112.07945/) (2021)
- [Nerfies: Deformable Neural Radiance Fields](https://mlsanity.com/paper/2011.12948/) (2020)
- [Alias-Free Generative Adversarial Networks](https://mlsanity.com/paper/2106.12423/) (2021)
- [Learning Implicit Fields for Generative Shape Modeling](https://mlsanity.com/paper/1812.02822/) (2018)
- [Volume Rendering of Neural Implicit Surfaces](https://mlsanity.com/paper/2106.12052/) (2021)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields](https://mlsanity.com/paper/2011.12100/) (2020)
- [pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis](https://mlsanity.com/paper/2012.00926/) (2020)
- [GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis](https://mlsanity.com/paper/2007.02442/) (2020)
- [Which Training Methods for GANs do actually Converge?](https://mlsanity.com/paper/1801.04406/) (2018)
- [Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans](https://mlsanity.com/paper/2012.15838/) (2020)
- [Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loop](https://mlsanity.com/paper/1909.12828/) (2019)
- [Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling](https://mlsanity.com/paper/1610.07584/) (2016)
- [PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization](https://mlsanity.com/paper/2004.00452/) (2020)
- [HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Video](https://mlsanity.com/paper/2201.04127/) (2022)
- [StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis](https://mlsanity.com/paper/2110.08985/) (2021)
- [Pivotal Tuning for Latent-based Editing of Real Images](https://mlsanity.com/paper/2106.05744/) (2021)
- [PARE: Part Attention Regressor for 3D Human Body Estimation](https://mlsanity.com/paper/2104.08527/) (2021)
- [StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generation](https://mlsanity.com/paper/2112.11427/) (2021)
- [ARCH: Animatable Reconstruction of Clothed Humans](https://mlsanity.com/paper/2004.04572/) (2020)
- [A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose](https://mlsanity.com/paper/2102.06199/) (2021)
- [GRAM: Generative Radiance Manifolds for 3D-Aware Image Generation](https://mlsanity.com/paper/2112.08867/) (2021)

## Built on by (most cited first, 30 of 59)

- [MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model](https://mlsanity.com/paper/2311.16498/) (2023)
- [ECON: Explicit Clothed humans Optimized via Normal integration](https://mlsanity.com/paper/2212.07422/) (2022)
- [InstantAvatar: Learning Avatars from Monocular Video in 60 Seconds](https://mlsanity.com/paper/2212.10550/) (2022)
- [TeCH: Text-guided Reconstruction of Lifelike Clothed Humans](https://mlsanity.com/paper/2308.08545/) (2023)
- [PanoHead: Geometry-Aware 3D Full-Head Synthesis in 360circ](https://mlsanity.com/paper/2303.13071/) (2023)
- [Advances in 3D Generation: A Survey](https://mlsanity.com/paper/2401.17807/) (2024)
- [Gaussian Shell Maps for Efficient 3D Human Generation](https://mlsanity.com/paper/2311.17857/) (2023)
- [Fast-SNARF: A Fast Deformer for Articulated Neural Fields](https://mlsanity.com/paper/2211.15601/) (2022)
- [AG3D: Learning to Generate 3D Avatars from 2D Image Collections](https://mlsanity.com/paper/2305.02312/) (2023)
- [Learning Locally Editable Virtual Humans](https://mlsanity.com/paper/2305.00121/) (2023)
- [Single-Image 3D Human Digitization with Shape-Guided Diffusion](https://mlsanity.com/paper/2311.09221/) (2023)
- [OmniAvatar: Geometry-Guided Controllable 3D Head Synthesis](https://mlsanity.com/paper/2303.15539/) (2023)
- [PrimDiffusion: Volumetric Primitives Diffusion for 3D Human Generation](https://mlsanity.com/paper/2312.04559/) (2023)
- [HumanGen: Generating Human Radiance Fields with Explicit Priors](https://mlsanity.com/paper/2212.05321/) (2022)
- [Disentangled Clothed Avatar Generation from Text Descriptions](https://mlsanity.com/paper/2312.05295/) (2023)
- [Human101: Training 100+FPS Human Gaussians in 100s from 1 View](https://mlsanity.com/paper/2312.15258/) (2023)
- [GETAvatar: Generative Textured Meshes for Animatable Human Avatars](https://mlsanity.com/paper/2310.02714/) (2023)
- [UrbanGIRAFFE: Representing Urban Scenes as Compositional Generative Neural Feature Fields](https://mlsanity.com/paper/2303.14167/) (2023)
- [AI-Generated Content (AIGC) for Various Data Modalities: A Survey](https://mlsanity.com/paper/2308.14177/) (2023)
- [PV3D: A 3D Generative Model for Portrait Video Generation](https://mlsanity.com/paper/2212.06384/) (2022)
- [HumanLiff: Layer-wise 3D Human Generation with Diffusion Model](https://mlsanity.com/paper/2308.09712/) (2023)
- [Chupa: Carving 3D Clothed Humans from Skinned Shape Priors using 2D Diffusion Probabilistic Models](https://mlsanity.com/paper/2305.11870/) (2023)
- [AvatarFusion: Zero-shot Generation of Clothing-Decoupled 3D Avatars Using 2D Diffusion](https://mlsanity.com/paper/2307.06526/) (2023)
- [Get3DHuman: Lifting StyleGAN-Human into a 3D Generative Model using Pixel-aligned Reconstruction Priors](https://mlsanity.com/paper/2302.01162/) (2023)
- [3DHumanGAN: 3D-Aware Human Image Generation with 3D Pose Mapping](https://mlsanity.com/paper/2212.07378/) (2022)
- [XAGen: 3D Expressive Human Avatars Generation](https://mlsanity.com/paper/2311.13574/) (2023)
- [StructLDM: Structured Latent Diffusion for 3D Human Generation](https://mlsanity.com/paper/2404.01241/) (2024)
- [AniPortraitGAN: Animatable 3D Portrait Generation from 2D Image Collections](https://mlsanity.com/paper/2309.02186/) (2023)
- [AvatarStudio: High-fidelity and Animatable 3D Avatar Creation from Text](https://mlsanity.com/paper/2311.17917/) (2023)
- [LumiGAN: Unconditional Generation of Relightable 3D Human Faces](https://mlsanity.com/paper/2304.13153/) (2023)

## Abstract (arXiv)

Unsupervised generation of 3D-aware clothed humans with various appearances and controllable geometries is important for creating virtual human avatars and other AR/VR applications. Existing methods are either limited to rigid object modeling, or not generative and thus unable to generate high-quality virtual humans and animate them. In this work, we propose AvatarGen, the first method that enables not only geometry-aware clothed human synthesis with high-fidelity appearances but also disentangled human animation controllability, while only requiring 2D images for training. Specifically, we decompose the generative 3D human synthesis into pose-guided mapping and canonical representation with predefined human pose and shape, such that the canonical representation can be explicitly driven to different poses and shapes with the guidance of a 3D parametric human model SMPL. AvatarGen further introduces a deformation network to learn non-rigid deformations for modeling fine-grained geometric details and pose-dependent dynamics. To improve the geometry quality of the generated human avatars, it leverages the signed distance field as geometric proxy, which allows more direct regularization from the 3D geometric priors of SMPL. Benefiting from these designs, our method can generate animatable 3D human avatars with high-quality appearance and geometry modeling, significantly outperforming previous 3D GANs. Furthermore, it is competent for many applications, e.g., single-view reconstruction, re-animation, and text-guided synthesis/editing. Code and pre-trained model will be available at http://jeff95.me/projects/avatargen.html.