# Modeling Clothing as a Separate Layer for an Animatable Human Avatar

arXiv: https://arxiv.org/abs/2106.14879 · HTML: https://mlsanity.com/paper/2106.14879/

## Key facts

- Cited by 66 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 88 references that are in the map.
- First posted on arXiv 28 Jun 2021.

## Builds on (most cited first, 30 of 36)

- [D-NeRF: Neural Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2011.13961/) (2020)
- [Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans](https://mlsanity.com/paper/2012.15838/) (2020)
- [Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning](https://mlsanity.com/paper/1904.01786/) (2019)
- [Video-to-Video Synthesis](https://mlsanity.com/paper/1808.06601/) (2018)
- [Everybody Dance Now](https://mlsanity.com/paper/1808.07371/) (2018)
- [STAR: Sparse Trained Articulated Human Body Regressor](https://mlsanity.com/paper/2008.08535/) (2020)
- [SCANimate: Weakly Supervised Learning of Skinned Clothed Avatar Networks](https://mlsanity.com/paper/2104.03313/) (2021)
- [Deep Appearance Models for Face Rendering](https://mlsanity.com/paper/1808.00362/) (2018)
- [DeepCap: Monocular Human Performance Capture Using Weak Supervision](https://mlsanity.com/paper/2003.08325/) (2020)
- [Liquid Warping GAN: A Unified Framework for Human Motion Imitation, Appearance Transfer and Novel View Synthesis](https://mlsanity.com/paper/1909.12224/) (2019)
- [TailorNet: Predicting Clothing in 3D as a Function of Human Pose, Shape and Garment Style](https://mlsanity.com/paper/2003.04583/) (2020)
- [Real-time Deep Dynamic Characters](https://mlsanity.com/paper/2105.01794/) (2021)
- [CLOTH3D: Clothed 3D Humans](https://mlsanity.com/paper/1912.02792/) (2019)
- [LiveCap: Real-time Human Performance Capture from Monocular Video](https://mlsanity.com/paper/1810.02648/) (2018)
- [Towards Accurate Markerless Human Shape and Pose Estimation over Time](https://mlsanity.com/paper/1707.07548/) (2017)
- [DeepWrinkles: Accurate and Realistic Clothing Modeling](https://mlsanity.com/paper/1808.03417/) (2018)
- [Detailed, accurate, human shape estimation from clothed 3D scan sequences](https://mlsanity.com/paper/1703.04454/) (2017)
- [Learning-Based Animation of Clothing for Virtual Try-On](https://mlsanity.com/paper/1903.07190/) (2019)
- [Self-Supervised Collision Handling via Generative 3D Garment Models for Virtual Try-On](https://mlsanity.com/paper/2105.06462/) (2021)
- [Driving-Signal Aware Full-Body Avatars](https://mlsanity.com/paper/2105.10441/) (2021)
- [Learning Compositional Radiance Fields of Dynamic Human Heads](https://mlsanity.com/paper/2012.09955/) (2020)
- [SCALE: Modeling Clothed Humans with a Surface Codec of Articulated Local Elements](https://mlsanity.com/paper/2104.07660/) (2021)
- [GarNet: A Two-Stream Network for Fast and Accurate 3D Cloth Draping](https://mlsanity.com/paper/1811.10983/) (2018)
- [SMPLpix: Neural Avatars from 3D Human Models](https://mlsanity.com/paper/2008.06872/) (2020)
- [PBNS: Physically Based Neural Simulator for Unsupervised Garment Pose Space Deformation](https://mlsanity.com/paper/2012.11310/) (2020)
- [MonoClothCap: Towards Temporally Coherent Clothing Capture from Monocular RGB Video](https://mlsanity.com/paper/2009.10711/) (2020)
- [Neural Re-Rendering of Humans from a Single Image](https://mlsanity.com/paper/2101.04104/) (2021)
- [Unsupervised Person Image Synthesis in Arbitrary Poses](https://mlsanity.com/paper/1809.10280/) (2018)
- [ANR: Articulated Neural Rendering for Virtual Avatars](https://mlsanity.com/paper/2012.12890/) (2020)
- [Fully Convolutional Graph Neural Networks for Parametric Virtual Try-On](https://mlsanity.com/paper/2009.04592/) (2020)

## Built on by (most cited first, 30 of 66)

- [MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model](https://mlsanity.com/paper/2311.16498/) (2023)
- [SplattingAvatar: Realistic Real-Time Human Avatars with Mesh-Embedded Gaussian Splatting](https://mlsanity.com/paper/2403.05087/) (2024)
- [Drivable 3D Gaussian Avatars](https://mlsanity.com/paper/2311.08581/) (2023)
- [HuMMan: Multi-Modal 4D Human Dataset for Versatile Sensing and Modeling](https://mlsanity.com/paper/2204.13686/) (2022)
- [Structured Local Radiance Fields for Human Avatar Modeling](https://mlsanity.com/paper/2203.14478/) (2022)
- [ASH: Animatable Gaussian Splats for Efficient and Photoreal Human Rendering](https://mlsanity.com/paper/2312.05941/) (2023)
- [AvatarReX: Real-time Expressive Full-body Avatars](https://mlsanity.com/paper/2305.04789/) (2023)
- [Capturing and Animation of Body and Clothing from Monocular Video](https://mlsanity.com/paper/2210.01868/) (2022)
- [Human Performance Modeling and Rendering via Neural Animated Mesh](https://mlsanity.com/paper/2209.08468/) (2022)
- [Drivable Volumetric Avatars using Texel-Aligned Features](https://mlsanity.com/paper/2207.09774/) (2022)
- [AvatarGen: A 3D Generative Model for Animatable Human Avatars](https://mlsanity.com/paper/2211.14589/) (2022)
- [PoseVocab: Learning Joint-structured Pose Embeddings for Human Avatar Modeling](https://mlsanity.com/paper/2304.13006/) (2023)
- [Learning Disentangled Avatars with Hybrid 3D Representations](https://mlsanity.com/paper/2309.06441/) (2023)
- [MeshAvatar: Learning High-quality Triangular Human Avatars from Multi-view Videos](https://mlsanity.com/paper/2407.08414/) (2024)
- [Vid2Avatar-Pro: Authentic Avatar from Videos in the Wild via Universal Prior](https://mlsanity.com/paper/2503.01610/) (2025)
- [TriHuman : A Real-time and Controllable Tri-plane Representation for Detailed Human Geometry and Appearance Synthesis](https://mlsanity.com/paper/2312.05161/) (2023)
- [Animatable and Relightable Gaussians for High-fidelity Human Avatar Modeling](https://mlsanity.com/paper/2311.16096/) (2023)
- [AvatarCap: Animatable Avatar Conditioned Monocular Human Volumetric Capture](https://mlsanity.com/paper/2207.02031/) (2022)
- [Gaussian Garments: Reconstructing Simulation-Ready Clothing with Photorealistic Appearance from Multi-View Video](https://mlsanity.com/paper/2409.08189/) (2024)
- [PERGAMO: Personalized 3D Garments from Monocular Video](https://mlsanity.com/paper/2210.15040/) (2022)
- [Drivable Avatar Clothing: Faithful Full-Body Telepresence with Dynamic Clothing Driven by Sparse RGB-D Input](https://mlsanity.com/paper/2310.05917/) (2023)
- [LayGA: Layered Gaussian Avatars for Animatable Clothing Transfer](https://mlsanity.com/paper/2405.07319/) (2024)
- [CloSET: Modeling Clothed Humans on Continuous Surface with Explicit Template Decomposition](https://mlsanity.com/paper/2304.03167/) (2023)
- [AvatarFusion: Zero-shot Generation of Clothing-Decoupled 3D Avatars Using 2D Diffusion](https://mlsanity.com/paper/2307.06526/) (2023)
- [CaPhy: Capturing Physical Properties for Animatable Human Avatars](https://mlsanity.com/paper/2308.05925/) (2023)
- [Dressing Avatars: Deep Photorealistic Appearance for Physically Simulated Clothing](https://mlsanity.com/paper/2206.15470/) (2022)
- [ReLoo: Reconstructing Humans Dressed in Loose Garments from Monocular Video in the Wild](https://mlsanity.com/paper/2409.15269/) (2024)
- [XAGen: 3D Expressive Human Avatars Generation](https://mlsanity.com/paper/2311.13574/) (2023)
- [Disco4D: Disentangled 4D Human Generation and Animation from a Single Image](https://mlsanity.com/paper/2409.17280/) (2024)
- [DressRecon: Freeform 4D Human Reconstruction from Monocular Video](https://mlsanity.com/paper/2409.20563/) (2024)

## Abstract (arXiv)

We have recently seen great progress in building photorealistic animatable full-body codec avatars, but generating high-fidelity animation of clothing is still difficult. To address these difficulties, we propose a method to build an animatable clothed body avatar with an explicit representation of the clothing on the upper body from multi-view captured videos. We use a two-layer mesh representation to register each 3D scan separately with the body and clothing templates. In order to improve the photometric correspondence across different frames, texture alignment is then performed through inverse rendering of the clothing geometry and texture predicted by a variational autoencoder. We then train a new two-layer codec avatar with separate modeling of the upper clothing and the inner body layer. To learn the interaction between the body dynamics and clothing states, we use a temporal convolution network to predict the clothing latent code based on a sequence of input skeletal poses. We show photorealistic animation output for three different actors, and demonstrate the advantage of our clothed-body avatars over the single-layer avatars used in previous work. We also show the benefit of an explicit clothing model that allows the clothing texture to be edited in the animation output.