# Neural 3D Clothes Retargeting from a Single Image

arXiv: https://arxiv.org/abs/2102.00062 · HTML: https://mlsanity.com/paper/2102.00062/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 70 references that are in the map.
- First posted on arXiv 29 Jan 2021.

## Builds on (most cited first, 30 of 30)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [End-to-end Recovery of Human Shape and Pose](https://mlsanity.com/paper/1712.06584/) (2017)
- [Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image](https://mlsanity.com/paper/1607.08128/) (2016)
- [Face2Face: Real-time Face Capture and Reenactment of RGB Videos](https://mlsanity.com/paper/2007.14808/) (2020)
- [DensePose: Dense Human Pose Estimation In The Wild](https://mlsanity.com/paper/1802.00434/) (2018)
- [RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentation](https://mlsanity.com/paper/1611.06612/) (2016)
- [Learning from Synthetic Humans](https://mlsanity.com/paper/1701.01370/) (2017)
- [VITON: An Image-based Virtual Try-on Network](https://mlsanity.com/paper/1711.08447/) (2017)
- [Unite the People: Closing the Loop Between 3D and 2D Human Representations](https://mlsanity.com/paper/1701.02468/) (2017)
- [MoFA: Model-based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction](https://mlsanity.com/paper/1703.10580/) (2017)
- [Monocular Total Capture: Posing Face, Body, and Hands in the Wild](https://mlsanity.com/paper/1812.01598/) (2018)
- [DeepCap: Monocular Human Performance Capture Using Weak Supervision](https://mlsanity.com/paper/2003.08325/) (2020)
- [Face Alignment in Full Pose Range: A 3D Total Solution](https://mlsanity.com/paper/1804.01005/) (2018)
- [Regressing Robust and Discriminative 3D Morphable Models with a very Deep Neural Network](https://mlsanity.com/paper/1612.04904/) (2016)
- [DenseRaC: Joint 3D Pose and Shape Estimation by Dense Render-and-Compare](https://mlsanity.com/paper/1910.00116/) (2019)
- [DoubleFusion: Real-time Capture of Human Performances with Inner Body Shapes from a Single Depth Sensor](https://mlsanity.com/paper/1804.06023/) (2018)
- [LiveCap: Real-time Human Performance Capture from Monocular Video](https://mlsanity.com/paper/1810.02648/) (2018)
- [Self-supervised Multi-level Face Model Learning for Monocular Reconstruction at over 250 Hz](https://mlsanity.com/paper/1712.02859/) (2017)
- [DeepWrinkles: Accurate and Realistic Clothing Modeling](https://mlsanity.com/paper/1808.03417/) (2018)
- [Learning-Based Animation of Clothing for Virtual Try-On](https://mlsanity.com/paper/1903.07190/) (2019)
- [Learning Detailed Face Reconstruction from a Single Image](https://mlsanity.com/paper/1611.05053/) (2016)
- [GarNet: A Two-Stream Network for Fast and Accurate 3D Cloth Draping](https://mlsanity.com/paper/1811.10983/) (2018)
- [Synthesizing Training Images for Boosting Human 3D Pose Estimation](https://mlsanity.com/paper/1604.02703/) (2016)
- [Be Your Own Prada: Fashion Synthesis with Structural Coherence](https://mlsanity.com/paper/1710.07346/) (2017)
- [Automatic Face Reenactment](https://mlsanity.com/paper/1602.02651/) (2016)
- [SimulCap : Single-View Human Performance Capture with Cloth Simulation](https://mlsanity.com/paper/1903.06323/) (2019)
- [Pixel-Level Matching for Video Object Segmentation using Convolutional Neural Networks](https://mlsanity.com/paper/1708.05137/) (2017)
- [Detailed Garment Recovery from a Single-View Image](https://mlsanity.com/paper/1608.01250/) (2016)
- [Self-Supervised Adaptation of High-Fidelity Face Models for Monocular Performance Tracking](https://mlsanity.com/paper/1907.10815/) (2019)
- [3D Semantic Trajectory Reconstruction from 3D Pixel Continuum](https://mlsanity.com/paper/1712.01359/) (2017)

## Built on by (most cited first, 5 of 5)

- [Point-Based Modeling of Human Clothing](https://mlsanity.com/paper/2104.08230/) (2021)
- [HUMBI: A Large Multiview Dataset of Human Body Expressions and Benchmark Challenge](https://mlsanity.com/paper/2110.00119/) (2021)
- [BodyMap: Learning Full-Body Dense Correspondence Map](https://mlsanity.com/paper/2205.09111/) (2022)
- [Smart Fashion: A Review of AI Applications in the Fashion & Apparel Industry](https://mlsanity.com/paper/2111.00905/) (2021)
- [fAshIon after fashion: A Report of AI in Fashion](https://mlsanity.com/paper/2105.03050/) (2021)

## Abstract (arXiv)

In this paper, we present a method of clothes retargeting; generating the potential poses and deformations of a given 3D clothing template model to fit onto a person in a single RGB image. The problem is fundamentally ill-posed as attaining the ground truth data is impossible, i.e., images of people wearing the different 3D clothing template model at exact same pose. We address this challenge by utilizing large-scale synthetic data generated from physical simulation, allowing us to map 2D dense body pose to 3D clothing deformation. With the simulated data, we propose a semi-supervised learning framework that validates the physical plausibility of the 3D deformation by matching with the prescribed body-to-cloth contact points and clothing silhouette to fit onto the unlabeled real images. A new neural clothes retargeting network (CRNet) is designed to integrate the semi-supervised retargeting task in an end-to-end fashion. In our evaluation, we show that our method can predict the realistic 3D pose and deformation field needed for retargeting clothes models in real-world examples.