# Learning Locally Editable Virtual Humans

arXiv: https://arxiv.org/abs/2305.00121 · HTML: https://mlsanity.com/paper/2305.00121/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- Builds on 61 of its 82 references that are in the map.
- First posted on arXiv 28 Apr 2023.

## Builds on (most cited first, 30 of 61)

- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [Alias-Free Generative Adversarial Networks](https://mlsanity.com/paper/2106.12423/) (2021)
- [PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization](https://mlsanity.com/paper/1905.05172/) (2019)
- [Which Training Methods for GANs do actually Converge?](https://mlsanity.com/paper/1801.04406/) (2018)
- [Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans](https://mlsanity.com/paper/2012.15838/) (2020)
- [Neural Fields in Visual Computing and Beyond](https://mlsanity.com/paper/2111.11426/) (2021)
- [PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization](https://mlsanity.com/paper/2004.00452/) (2020)
- [AI Choreographer: Music Conditioned 3D Dance Generation with AIST++](https://mlsanity.com/paper/2101.08779/) (2021)
- [One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing](https://mlsanity.com/paper/2011.15126/) (2020)
- [Pivotal Tuning for Latent-based Editing of Real Images](https://mlsanity.com/paper/2106.05744/) (2021)
- [Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Shapes](https://mlsanity.com/paper/2101.10994/) (2021)
- [GAN Inversion: A Survey](https://mlsanity.com/paper/2101.05278/) (2021)
- [Advances in Neural Rendering](https://mlsanity.com/paper/2111.05849/) (2021)
- [Learning from Synthetic Humans](https://mlsanity.com/paper/1701.01370/) (2017)
- [Everybody Dance Now](https://mlsanity.com/paper/1808.07371/) (2018)
- [State of the Art on Neural Rendering](https://mlsanity.com/paper/2004.03805/) (2020)
- [Deferred Neural Rendering: Image Synthesis using Neural Textures](https://mlsanity.com/paper/1904.12356/) (2019)
- [Function4D: Real-time Human Volumetric Capture from Very Sparse Consumer RGBD Sensors](https://mlsanity.com/paper/2105.01859/) (2021)
- [I M Avatar: Implicit Morphable Head Avatars from Videos](https://mlsanity.com/paper/2112.07471/) (2021)
- [SNARF: Differentiable Forward Skinning for Animating Non-Rigid Neural Implicit Shapes](https://mlsanity.com/paper/2104.03953/) (2021)
- [AGORA: Avatars in Geography Optimized for Regression Analysis](https://mlsanity.com/paper/2104.14643/) (2021)
- [STAR: Sparse Trained Articulated Human Body Regressor](https://mlsanity.com/paper/2008.08535/) (2020)
- [SCANimate: Weakly Supervised Learning of Skinned Clothed Avatar Networks](https://mlsanity.com/paper/2104.03313/) (2021)
- [Embodied Hands: Modeling and Capturing Hands and Bodies Together](https://mlsanity.com/paper/2201.02610/) (2022)
- [Learning to Reconstruct People in Clothing from a Single RGB Camera](https://mlsanity.com/paper/1903.05885/) (2019)

## Built on by (most cited first, 30 of 54)

- [Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation](https://mlsanity.com/paper/2311.17117/) (2023)
- [SplattingAvatar: Realistic Real-Time Human Avatars with Mesh-Embedded Gaussian Splatting](https://mlsanity.com/paper/2403.05087/) (2024)
- [HumanNorm: Learning Normal Diffusion Model for High-quality and Realistic 3D Human Generation](https://mlsanity.com/paper/2310.01406/) (2023)
- [SiTH: Single-view Textured Human Reconstruction with Image-Conditioned Diffusion](https://mlsanity.com/paper/2311.15855/) (2023)
- [4D-DRESS: A 4D Dataset of Real-world Human Clothing with Semantic Annotations](https://mlsanity.com/paper/2404.18630/) (2024)
- [Human-3Diffusion: Realistic Avatar Creation via Explicit 3D Consistent Diffusion Models](https://mlsanity.com/paper/2406.08475/) (2024)
- [AniGS: Animatable Gaussian Avatar from a Single Image with Inconsistent Gaussian Reconstruction](https://mlsanity.com/paper/2412.02684/) (2024)
- [MeshAvatar: Learning High-quality Triangular Human Avatars from Multi-view Videos](https://mlsanity.com/paper/2407.08414/) (2024)
- [Generalizable Human Gaussians from Single-View Image](https://mlsanity.com/paper/2406.06050/) (2024)
- [GarmentCodeData: A Dataset of 3D Made-to-Measure Garments With Sewing Patterns](https://mlsanity.com/paper/2405.17609/) (2024)
- [Animatable and Relightable Gaussians for High-fidelity Human Avatar Modeling](https://mlsanity.com/paper/2311.16096/) (2023)
- [PSHuman: Photorealistic Single-image 3D Human Reconstruction using Cross-Scale Multiview Diffusion and Explicit Remeshing](https://mlsanity.com/paper/2409.10141/) (2024)
- [UV Gaussians: Joint Learning of Mesh Deformation and Gaussian Textures for Human Avatar Modeling](https://mlsanity.com/paper/2403.11589/) (2024)
- [Gen-3Diffusion: Realistic Image-to-3D Generation via 2D & 3D Diffusion Synergy](https://mlsanity.com/paper/2412.06698/) (2024)
- [PERSE: Personalized 3D Generative Avatars from A Single Portrait](https://mlsanity.com/paper/2412.21206/) (2024)
- [Simulating the Real World: A Unified Survey of Multimodal Generative Models](https://mlsanity.com/paper/2503.04641/) (2025)
- [HumanDreamer-X: Photorealistic Single-image Human Avatars Reconstruction via Gaussian Restoration](https://mlsanity.com/paper/2504.03536/) (2025)
- [Diffusion Models in 3D Vision: A Survey](https://mlsanity.com/paper/2410.04738/) (2024)
- [DEGAS: Detailed Expressions on Full-Body Gaussian Avatars](https://mlsanity.com/paper/2408.10588/) (2024)
- [MultiGO: Towards Multi-level Geometry Learning for Monocular 3D Textured Human Reconstruction](https://mlsanity.com/paper/2412.03103/) (2024)
- [UP2You: Fast Reconstruction of Yourself from Unconstrained Photo Collections](https://mlsanity.com/paper/2509.24817/) (2025)
- [SyncHuman: Synchronizing 2D and 3D Generative Models for Single-view Human Reconstruction](https://mlsanity.com/paper/2510.07723/) (2025)
- [AdaHuman: Animatable Detailed 3D Human Generation with Compositional Multiview Diffusion](https://mlsanity.com/paper/2505.24877/) (2025)
- [MoGA: 3D Generative Avatar Prior for Monocular Gaussian Avatar Reconstruction](https://mlsanity.com/paper/2507.23597/) (2025)
- [PEGASUS: Personalized Generative 3D Avatars with Composable Attributes](https://mlsanity.com/paper/2402.10636/) (2024)
- [Disentangled Clothed Avatar Generation with Layered Representation](https://mlsanity.com/paper/2501.04631/) (2025)
- [HumanGif: Single-View Human Diffusion with Generative Prior](https://mlsanity.com/paper/2502.12080/) (2025)
- [Joint2Human: High-quality 3D Human Generation via Compact Spherical Embedding of 3D Joints](https://mlsanity.com/paper/2312.08591/) (2023)
- [GeneMAN: Generalizable Single-Image 3D Human Reconstruction from Multi-Source Human Data](https://mlsanity.com/paper/2411.18624/) (2024)
- [CHROME: Clothed Human Reconstruction with Occlusion-Resilience and Multiview-Consistency from a Single Image](https://mlsanity.com/paper/2503.15671/) (2025)

## Abstract (arXiv)

In this paper, we propose a novel hybrid representation and end-to-end trainable network architecture to model fully editable and customizable neural avatars. At the core of our work lies a representation that combines the modeling power of neural fields with the ease of use and inherent 3D consistency of skinned meshes. To this end, we construct a trainable feature codebook to store local geometry and texture features on the vertices of a deformable body model, thus exploiting its consistent topology under articulation. This representation is then employed in a generative auto-decoder architecture that admits fitting to unseen scans and sampling of realistic avatars with varied appearances and geometries. Furthermore, our representation allows local editing by swapping local features between 3D assets. To verify our method for avatar creation and editing, we contribute a new high-quality dataset, dubbed CustomHumans, for training and evaluation. Our experiments quantitatively and qualitatively show that our method generates diverse detailed avatars and achieves better model fitting performance compared to state-of-the-art methods. Our code and dataset are available at https://custom-humans.github.io/.