Garment Avatars: Realistic Cloth Driving using Pattern Registration

arXiv 2206.03373 ↗First posted 7 Jun 2022cs.CVcited by 6

Not on the map. Why?

Abstract (arXiv)

Virtual telepresence is the future of online communication. Clothing is an essential part of a person's identity and self-expression. Yet, ground truth data of registered clothes is currently unavailable in the required resolution and accuracy for training telepresence models for realistic cloth animation. Here, we propose an end-to-end pipeline for building drivable representations for clothing. The core of our approach is a multi-view patterned cloth tracking algorithm capable of capturing deformations with high accuracy. We further rely on the high-quality data produced by our tracking method to build a Garment Avatar: an expressive and fully-drivable geometry model for a piece of clothing. The resulting model can be animated using a sparse set of views and produces highly realistic reconstructions which are faithful to the driving signals. We demonstrate the efficacy of our pipeline on a realistic virtual telepresence application, where a garment is being reconstructed from two views, and a user can pick and swap garment design as they wish. In addition, we show a challenging scenario when driven exclusively with body pose, our drivable garment avatar is capable of producing realistic cloth geometry of significantly higher quality than the state-of-the-art.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 44 of its 90 references that are in the map. First posted on arXiv 7 Jun 2022.

Builds on

44 of 90 references.

References · 44

2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
2019PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitizationcited by 818
2020Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humanscited by 634
2020PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitizationcited by 521
2021Function4D: Real-time Human Volumetric Capture from Very Sparse Consumer RGBD Sensorscited by 246
2019Multi-Garment Net: Learning to Dress 3D People from Imagescited by 244
2021SCANimate: Weakly Supervised Learning of Skinned Clothed Avatar Networkscited by 181
2019Tex2Shape: Detailed Full Human Body Geometry From a Single Imagecited by 166
2018Deep Appearance Models for Face Renderingcited by 164
2020DeepCap: Monocular Human Performance Capture Using Weak Supervisioncited by 162
2017Deep Functional Maps: Structured Prediction for Dense Shape Correspondencecited by 134

32 more

2020Combining Implicit Function Learning and Parametric Models for 3D Human Reconstructioncited by 133
2021Real-time Deep Dynamic Characterscited by 133
2020LoopReg: Self-supervised Learning of Implicit Surface Correspondences, Pose and Shape for 3D Human Mesh Registrationcited by 108
2018DoubleFusion: Real-time Capture of Human Performances with Inner Body Shapes from a Single Depth Sensorcited by 95
2018DeepWrinkles: Accurate and Realistic Clothing Modelingcited by 90
2021The Power of Points for Modeling Humans in Clothingcited by 90
2017Detailed, accurate, human shape estimation from clothed 3D scan sequencescited by 89
2019Learning-Based Animation of Clothing for Virtual Try-Oncited by 89
2018Shift-Net: Image Inpainting via Deep Feature Rearrangementcited by 87
2020SIZER: A Dataset and Model for Parsing 3D Clothing and Learning Size Sensitive 3D Clothingcited by 87
2021Self-Supervised Collision Handling via Generative 3D Garment Models for Virtual Try-Oncited by 81
2021Driving-Signal Aware Full-Body Avatarscited by 81
2020Monocular Real-Time Volumetric Performance Capturecited by 74
2020Deep Shells: Unsupervised Shape Correspondence with Optimal Transportcited by 74
2019Smooth Shells: Multi-Scale Shape Registration with Functional Mapscited by 72
2021SCALE: Modeling Clothed Humans with a Surface Codec of Articulated Local Elementscited by 72
2018SiCloPe: Silhouette-Based Clothed Peoplecited by 68
2021Modeling Clothing as a Separate Layer for an Animatable Human Avatarcited by 66
2017MonoPerfCap: Human Performance Capture from Monocular Videocited by 63
2020PBNS: Physically Based Neural Simulator for Unsupervised Garment Pose Space Deformationcited by 63
2020MonoClothCap: Towards Temporally Coherent Clothing Capture from Monocular RGB Videocited by 62
2021DPFM: Deep Partial Functional Mapscited by 58
2020LIMP: Learning Latent Shape Representations with Metric Preservation Priorscited by 54
2020Fully Convolutional Mesh Autoencoder using Efficient Spatially Varying Kernelscited by 53
2021GarmentNets: Category-Level Pose Estimation for Garments via Canonical Space Shape Completioncited by 41
2019Shape Reconstruction by Learning Differentiable Surface Representationscited by 37
2020Deep Physics-aware Inference of Cloth Deformation for Monocular Human Performance Capturecited by 30
2018A Pixel-Based Framework for Data-Driven Clothingcited by 28
2017Boundary First Flatteningcited by 17
2020Cloth in the Wind: A Case Study of Physical Measurement through Simulationcited by 9
2021Capturing Detailed Deformations of Moving Human Bodiescited by 5
2018Self Functional Mapscited by 4

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 3; 2024: 2.

6 papers.

2023Learning Disentangled Avatars with Hybrid 3D Representationscited by 44
2023REC-MV: REconstructing 3D Dynamic Cloth from Monocular Videoscited by 29
2024Disco4D: Disentangled 4D Human Generation and Animation from a Single Imagecited by 17
2022Relightable Neural Human Assets from Multi-view Gradient Illuminationscited by 11
2023High-Quality Animatable Dynamic Garment Reconstruction from Monocular Videoscited by 5
2024Pattern Guided UV Recovery for Realistic Video Garment Texturingcited by 1

Similar, not cited

Close in wording, no citation either way