# Structured 3D Features for Reconstructing Controllable Avatars

arXiv: https://arxiv.org/abs/2212.06820 · HTML: https://mlsanity.com/paper/2212.06820/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 71 of its 86 references that are in the map.
- First posted on arXiv 13 Dec 2022.

## Builds on (most cited first, 30 of 71)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [Learning Implicit Fields for Generative Shape Modeling](https://mlsanity.com/paper/1812.02822/) (2018)
- [Volume Rendering of Neural Implicit Surfaces](https://mlsanity.com/paper/2106.12052/) (2021)
- [End-to-end Recovery of Human Shape and Pose](https://mlsanity.com/paper/1712.06584/) (2017)
- [PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization](https://mlsanity.com/paper/1905.05172/) (2019)
- [Implicit Geometric Regularization for Learning Shapes](https://mlsanity.com/paper/2002.10099/) (2020)
- [Which Training Methods for GANs do actually Converge?](https://mlsanity.com/paper/1801.04406/) (2018)
- [Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans](https://mlsanity.com/paper/2012.15838/) (2020)
- [Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loop](https://mlsanity.com/paper/1909.12828/) (2019)
- [PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization](https://mlsanity.com/paper/2004.00452/) (2020)
- [HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Video](https://mlsanity.com/paper/2201.04127/) (2022)
- [Advances in Neural Rendering](https://mlsanity.com/paper/2111.05849/) (2021)
- [Animatable Neural Radiance Fields for Modeling Dynamic Human Bodies](https://mlsanity.com/paper/2105.02872/) (2021)
- [ICON: Implicit Clothed humans Obtained from Normals](https://mlsanity.com/paper/2112.09127/) (2021)
- [Convolutional Mesh Regression for Single-Image Human Shape Reconstruction](https://mlsanity.com/paper/1905.03244/) (2019)
- [ARCH: Animatable Reconstruction of Clothed Humans](https://mlsanity.com/paper/2004.04572/) (2020)
- [Multi-Garment Net: Learning to Dress 3D People from Images](https://mlsanity.com/paper/1908.06903/) (2019)
- [NeuMan: Neural Human Radiance Field from a Single Video](https://mlsanity.com/paper/2203.12575/) (2022)
- [Video Based Reconstruction of 3D People Models](https://mlsanity.com/paper/1803.04758/) (2018)
- [PaMIR: Parametric Model-Conditioned Implicit Representation for Image-based Human Reconstruction](https://mlsanity.com/paper/2007.03858/) (2020)
- [I M Avatar: Implicit Morphable Head Avatars from Videos](https://mlsanity.com/paper/2112.07471/) (2021)
- [Photographic Image Synthesis with Cascaded Refinement Networks](https://mlsanity.com/paper/1707.09405/) (2017)
- [Unite the People: Closing the Loop Between 3D and 2D Human Representations](https://mlsanity.com/paper/1701.02468/) (2017)

## Built on by (most cited first, 19 of 19)

- [TeCH: Text-guided Reconstruction of Lifelike Clothed Humans](https://mlsanity.com/paper/2308.08545/) (2023)
- [IDOL: Instant Photorealistic 3D Human Creation from a Single Image](https://mlsanity.com/paper/2412.14963/) (2024)
- [AniGS: Animatable Gaussian Avatar from a Single Image with Inconsistent Gaussian Reconstruction](https://mlsanity.com/paper/2412.02684/) (2024)
- [DINAR: Diffusion Inpainting of Neural Textures for One-Shot Human Avatars](https://mlsanity.com/paper/2303.09375/) (2023)
- [CVTHead: One-shot Controllable Head Avatar with Vertex-feature Transformer](https://mlsanity.com/paper/2311.06443/) (2023)
- [HumanRef: Single Image to 3D Human Generation via Reference-Guided Diffusion](https://mlsanity.com/paper/2311.16961/) (2023)
- [ConTex-Human: Free-View Rendering of Human from a Single Image with Texture-Consistent Synthesis](https://mlsanity.com/paper/2311.17123/) (2023)
- [Implicit Shape and Appearance Priors for Few-Shot Full Head Reconstruction](https://mlsanity.com/paper/2310.08784/) (2023)
- [Human as Points: Explicit Point-based 3D Human Reconstruction from Single-view RGB Images](https://mlsanity.com/paper/2311.02892/) (2023)
- [Semantic Human Mesh Reconstruction with Textures](https://mlsanity.com/paper/2403.02561/) (2024)
- [Human-VDM: Learning Single-Image 3D Human Gaussian Splatting from Video Diffusion Models](https://mlsanity.com/paper/2409.02851/) (2024)
- [DeClotH: Decomposable 3D Cloth and Human Body Reconstruction from a Single Image](https://mlsanity.com/paper/2503.19373/) (2025)
- [NECA: Neural Customizable Human Avatar](https://mlsanity.com/paper/2403.10335/) (2024)
- [PARTE: Part-Guided Texturing for 3D Human Reconstruction from a Single Image](https://mlsanity.com/paper/2507.17332/) (2025)
- [COSMU: Complete 3D human shape from monocular unconstrained images](https://mlsanity.com/paper/2407.10586/) (2024)
- [MVD-HuGaS: Human Gaussians from a Single Image via 3D Human Multi-view Diffusion Prior](https://mlsanity.com/paper/2503.08218/) (2025)
- [GLVD: Guided Learned Vertex Descent](https://mlsanity.com/paper/2510.06046/) (2025)
- [HumanNOVA: Photorealistic, Universal and Rapid 3D Human Avatar Modeling from a Single Image](https://mlsanity.com/paper/2606.02573/) (2026)
- [DiGS-Avatar: Single-Image Animatable 3D Human Reconstruction via UV-Space Diffusion](https://mlsanity.com/paper/2608.20759/) (2026)

## Abstract (arXiv)

We introduce Structured 3D Features, a model based on a novel implicit 3D representation that pools pixel-aligned image features onto dense 3D points sampled from a parametric, statistical human mesh surface. The 3D points have associated semantics and can move freely in 3D space. This allows for optimal coverage of the person of interest, beyond just the body shape, which in turn, additionally helps modeling accessories, hair, and loose clothing. Owing to this, we present a complete 3D transformer-based attention framework which, given a single image of a person in an unconstrained pose, generates an animatable 3D reconstruction with albedo and illumination decomposition, as a result of a single end-to-end model, trained semi-supervised, and with no additional postprocessing. We show that our S3F model surpasses the previous state-of-the-art on various tasks, including monocular 3D reconstruction, as well as albedo and shading estimation. Moreover, we show that the proposed methodology allows novel view synthesis, relighting, and re-posing the reconstruction, and can naturally be extended to handle multiple input images (e.g. different views of a person, or the same view, in different poses, in video). Finally, we demonstrate the editing capabilities of our model for 3D virtual try-on applications.